{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Loss Functions\n",
    "\n",
    "This python script illustrates the different loss functions for regression and classification.\n",
    "\n",
    "We start by loading the ncessary libraries and resetting the computational graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import tensorflow as tf\n",
    "from tensorflow.python.framework import ops\n",
    "ops.reset_default_graph()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Create a Graph Session"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "sess = tf.Session()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Numerical Predictions\n",
    "\n",
    "---------------------------------\n",
    "\n",
    "To start with our investigation of loss functions, we begin by looking at numerical loss functions.  To do so, we must create a sequence of predictions around a target.  For this exercise, we consider the target to be zero."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Various Predicted X-values\n",
    "x_vals = tf.linspace(-1., 1., 500)\n",
    "\n",
    "# Create our target of zero\n",
    "target = tf.constant(0.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### L2 Loss\n",
    "\n",
    "The L2 loss is one of the most common regression loss functions.  Here we show how to create it in TensorFlow and we evaluate it for plotting later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# L2 loss\n",
    "# L = (pred - actual)^2\n",
    "l2_y_vals = tf.square(target - x_vals)\n",
    "l2_y_out = sess.run(l2_y_vals)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### L1 Loss\n",
    "\n",
    "An alternative loss function to consider is the L1 loss. This is very similar to L2 except that we take the `absolute value` of the difference instead of squaring it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# L1 loss\n",
    "# L = abs(pred - actual)\n",
    "l1_y_vals = tf.abs(target - x_vals)\n",
    "l1_y_out = sess.run(l1_y_vals)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Pseudo-Huber Loss\n",
    "\n",
    "The psuedo-huber loss function is a smooth approximation to the L1 loss as the (predicted - target) values get larger.  When the predicted values are close to the target, the pseudo-huber loss behaves similar to the L2 loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# L = delta^2 * (sqrt(1 + ((pred - actual)/delta)^2) - 1)\n",
    "\n",
    "# Pseudo-Huber with delta = 0.25\n",
    "delta1 = tf.constant(0.25)\n",
    "phuber1_y_vals = tf.multiply(tf.square(delta1), tf.sqrt(1. + tf.square((target - x_vals)/delta1)) - 1.)\n",
    "phuber1_y_out = sess.run(phuber1_y_vals)\n",
    "\n",
    "# Pseudo-Huber with delta = 5\n",
    "delta2 = tf.constant(5.)\n",
    "phuber2_y_vals = tf.multiply(tf.square(delta2), tf.sqrt(1. + tf.square((target - x_vals)/delta2)) - 1.)\n",
    "phuber2_y_out = sess.run(phuber2_y_vals)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Plot the Regression Losses\n",
    "\n",
    "Here we use Matplotlib to plot the L1, L2, and Pseudo-Huber Losses."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAEACAYAAABVtcpZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd4VNXWwOHfTugiRWlKRxCEK5DQQSBIFRG5CApIUy6X\na0EsqAgqcLkKiPiJgIWmSDE2LHREDKD0EoHQS0IRCBASSnpmf3/sECmTOuWcZNb7PPOYmZw5e2Uc\nZs3ZZW2ltUYIIYRv8rM6ACGEENaRJCCEED5MkoAQQvgwSQJCCOHDJAkIIYQPkyQghBA+TJKAEEL4\nMEkCQgjhwyQJCCGED5MkIIQQPkySgBBC+LB8VgdwPaWUFDISQogc0FqrnDzPdlcCWmuXbg6Hg38G\n/5OTMSddPpc3bg6HpkoVzc6d7j/36NGjXTuHw4EOCEAvWWL562SHm8uvZ164/fvf6LfftuXrOXas\nZuhQG7xGFtxcYbsk4CqlFCNbjqRs0bJWh5IlSkGfPjB/vtWROKEUvPIKTJ5sdSTCLipVgueeszqK\nW2ht/g316WN1JFmTmJLIsOXDSExJtDqUvJcEABre3ZB8frbq6cpQv36wYAEkJ1sdiROPPw6PPgoO\nh9WRCDsYNQrKlLE6ilts3my+szRpYnUkWZPiSKFx+cYU8C9gdSh5MwmA6VbafXa31WFkSa1a5gvW\n6tXuPW9QUJDrJ8mfH4YNA788+1bJMre8niKNO1/PL7+E/v1NIsgNCucvzJN1n7Q6DACUq/1J7qSU\n0u6K50riFbos7MKKvisolK+QW87pSdOnwx9/wMKFVkciRO6SkADly8P27VC5stXRZC4mPobihYq7\n9ZxKKXReGRh2l6IFihIyMCRXJACAJ56AZcvg0iWrIxEid1m6FO6/P3ckAK01bea24XjMcatDSZNn\nk0BuU6oUtGkD331ndSRC3OTiRTPyalPXuoJyA6UUm/61iUrFK1kdSpo8nwTCo8N5eeXLVoeRJf37\nmze0baWkwLFjVkchvO2f/4TFi62Owqnz5yEkBB57zOpIss4Og8HXy/NJ4O7b7yaoSpDLc2m9oXNn\n2LMHwsOtjiQdf/4JQUE2ncYkPGL7djh6FB56yOpInAoOhocfhmLFrI4kc5/v/JyjF49aHcYt8nwS\nKOBfgK41u6JywbSBggXN2IAt1wwABAaajlfps/IdkyfDCy+YWWI2lJu6gpIdyRTOV9jqMG6RZ2cH\n3cyhHZy9cpa7br/LI+d3l82bzbqBAwdsOt3t559h3DjYssWmAQq3OX4c6tc3XYDF3TubxR327YO2\nbU2Y+XLPsiCPkNlBWbDy8EpGrRlldRiZatzY/HfzZmvjSFeXLmYK0/r1VkciPO2jj2DgQFsmAIB5\n88wKYbsngITkBKtDyJDPXAlcO29u6Bb63//gr7/g44+tjiQdn35q5rP+/LPVkQhPmj0b2rWz5dxL\nhwOqVIElS6BuXaujydhzS5+jZeWW9PpHL4+14cqVgM8kgdwkPBwaNoRTp8w4ge3ExsKiRdC3r9WR\nCB+1Zg28/DKEhlodSeYSUxJxaIdH1yxJd1A2rA1fy5d/2nkepvmGU7eujb9oFykiCUBY6vPPTU9V\nblDAv4CtF636XBIoV7QcVUtUtTqMTA0aZK7GhRA3io42yxbs/j1ky6ktbDyx0eowMuVzSaBmqZq0\nrNzS6jAy1b07bN0KJ05YHYkQ9hIcDO3bm1X2dnYx7iJRcVFWh5Epnx0TSExJJC4pzu2FnNzp2Wfh\nrrvgrbesjkT4jMhIKFnStusCABo1MrOUO3WyOhL7kDGBHJi8YTKfh35udRgZevpp0/dp61L+cXGw\nf7/VUQh3+de/bLxaEXbtgtOnzZWAXSU7knNFhYJrfPZKIMWRgr+fv1fayimtzVqd//s/ePBBq6NJ\nx2+/mUuWsDDZcyC3O3AAWrUy09MK229lK8CLL0LRomYatV3N2jGLI1FHGN9uvNfalCmiediUKWZs\nwLZfzrSGBg3M9fnDD1sdjXDFf/4DZcvC2LFWR+JUQgJUqACbNsE991gdTfoc2sGlhEuUKFTCa21K\nEnDBwt0LqVKiCs0rNvdqu1l14YJ5w4eHQwnvvaeyZ8ECM5VpzRqrIxE5de4c3HuvuRqw4faRAN9+\naxZQ/vab1ZHYj4wJuKDMbWUoXtC+g8N33gkdOsBXX1kdSQYefxwOHYIdO6yOROTUxx9Dz562TQAA\nc+aYqdN2dTjqMEsPLrU6jGzz+STQrlo76pSpY3UYGbL9moH8+U2lycmTrY5E5FSDBvDqq1ZHka4T\nJ0w9re7drY4kfTHxMZyLPWd1GNnm891B11xJvIK/8qdwfvsNiKWkQNWqZoFMvXpWR5OOmBjzr7RD\nB6sjEXnQ//5nyqh88onVkdiTdAe5wfBVw1lxeIXVYTjl72+WyM+ZY3UkGSheXBKA8AiHw0yVfvpp\nqyNxTmtNsiP3brQkVwKpkh3J5POzb03aa0XlTpyw7ew9ITxi9Wp45RVTLM6ORYDXHFvDjO0zCO4R\nbFkMMjvIRzz0EPTunXt2UhLCHXr0MOtknn3W6kic01oTFRfFnUXutCwG6Q5yo2lbprH9r+1Wh+HU\nkCHw2WdWRyHyjL/+gsuXrY4iQ2fOwK+/2rtYnFLK0gTgKkkCN6lVqpZt/4d26QIREbB7t9WRZOL8\n+VwQpGD4cJg50+ooMjRnjrkSsONG8qcunWLen/OsDsNlkgRu0q5aO6qUqGJ1GE7ly2emi9r+auCP\nP+Cpp8xqYmFPx4/DihW2nnifkgIzZpiFzHZ0JfEKiSmJVofhMrckAaVUJ6XUfqXUQaXU605+P0Qp\ntUsptVMptU4pVcsd7XrSuavnbLk36L/+ZRaOXb1qdSQZeOQRsw/xunVWRyLSY/P9gwFWrTLlohs0\nsDoS52qWqsmgQPsm0axyOQkopfyAaUBHoA7Q28mH/AKtdV2tdQAwCfg/V9v1tJdXvcyGExusDuMW\nFStCixbw9ddWR5IBPz+z958sHrOnS5fMnMthw6yOJEOffWbPqwCHdhCbFGt1GG7jjiuBxsAhrXWE\n1joJCAYevf4ArfWV6+4WBexcHBmAud3m0qZqG6vDcGrIELPXu631728qfR04YHUk4mazZpk1HTbc\nQP6akyfNhWQvz+3NnmPrI9bTd5GNR6qzyR1JoDxw/f5XJ1Mfu4FS6lml1GFgAvCCG9r1KD9l3+GS\nTp3g7FnYudPqSDJQpIj5Gvd/tr/o8z3Nm8Pbb1sdRYZmzzYJoGhRqyO5VesqrZnf3a5lfbPPHZ90\nzuam3jIiqLX+WGtdHXgdyDV7ZY1fP54/jv9hdRg38PeHwYNzwQDx0KHmskXYS9OmcN99VkeRruRk\nc7Fi57dOkfxFrA7BbdyxRPYkUOm6+xWAvzI4/msg3c6MMWPGpP0cFBREUFCQa9G5qE3VNlQrWc3S\nGJx5+mmoUwcmTYLbb7c6mnSULm1uQmTDsmVm3wC71ck6dvEYSw8t5fnGz1sdCiEhIYSEhLjlXC6v\nGFZK+QMHgLbAaWAL0Ftrve+6Y6prrQ+n/vwI8JbWurGTc8mK4Wx47DGzzZ4dB8+EyKnOnU118oED\nrY7kRuHR4Ww5tYXH6zxudSi3sLxshFKqEzAF0700W2s9QSk1FtiqtV6ilPoQaAckAheB569PEted\nx7ZJIDw6nNsL3G6rhWRr1pgKzrt327OmihDZdfiwGbKIiJAaWdlheRJwFzsngZG/juSBSg/QuUZn\nq0NJozX84x8wbRq0sedEJmEXp06Z/5a/Zc6Grbz0EhQsCBMmWB3J3+KT47maeNVWXwBvJknAh33y\nCfzyCyxaZHUkmTh6FK5cgbp1rY7ENw0ZAuXK2Xb/YDBvj8qVzQZ1dpq9uuzQMhbtW8SsrrOsDiVd\nkgR8mF3/4dxC9iG2Ti7YPxjs/YXGoR22njYuVUS9aMLvE1i4e6HVYaQpWtSsy7L9jkuyD7F1csH+\nwVqbbs2hQ62OxDk7JwBXyZVANh29eJTSRUpze0H7zMs8fBiaNTM1wWw9mDZpktkZZMECqyPxHXFx\nUKUKrF0LtexbssuOkxyWHFxCeHS4LaaEZkauBLyoWslqtkoAANWrQ6NGprCcrQ0eDMuXm+3RhHfM\nm2feHDZOAABTp8Lzz9snAQAE3hXIA5UesDoMj5MrgRzaE7mHpJQkAu4KsDoUwHy2jhxpelvs9A/p\nFi+/bJY8T5pkdSS+4cABswS3Th2rI0lXRAQEBpr/2rFMRG4gA8MWWLRvEQ7toEftHlaHApjNuGvV\nMptwPGDnLy9nz0JCAlSqlPmxwie8/jokJcEHH1gdiRERHYG/nz8VilWwOpQskyQgAJgyBTZuhGDr\n9rsWIlvi4sz3gU2b4J57rI7GWLBrARfjL+aKsYBrJAlYLC4pjsL5rR+RjYmBqlXN4JrN1wQJAZhZ\nwz/8AEuWWB1J7iYDwxYK3hPM8FXDrQ4DMJtE9e1rBtmEsDutTaXxF1+0OhLDoW2/zYlHyJWAixJT\nEtFaUzBfQatDAczC3MaNITxcBtl81unTcOGCqSliYytWmPGA0FB7TGYYtnwYLSu3tM04X3ZId5C4\nQY8e0KqVmXdta9u3Q/78UkrC3YYPN1+zbb69Z/v20K+fWexoBxfjLlLAvwC3FbjN6lCyTZKADew/\nv5+fD/zMay1eszoUNm6EJ580C3T9/a2OJgOffmrmtv70k9WR5B2XLpmBoZ07bT0D688/TcnoY8eg\nQAGro8n9ZEzABsreVpZ7StpjekOzZqZW2A8/WB1JJvr3NxlL9iF2n2v7B9s4AYCZDjp0qD0SwOc7\nPyc8OtzqMCwjScBNShYuyWO1H7M6jDSvvGL73gDZh9jdkpLMPOFXXrE6kgydOgWLF9tn+8j45HgK\n5StkdRiWke4gN9Nas+P0Dhrc3cDSOFJSTOHIefPMJh22dfasWeV28KBsRemqr74yXWxr11odSYZG\njIDYWPjoI6sjyTukO8hGzsee563f3iLZkWxpHP7+Zuqd7a8GypY1I9mffWZ1JLnfgw/avpzslSum\nx8oO00KPxxy3OgRbkCuBPOzKFVNAcvNm+6zGdOrsWbOdVIkSVkciPOyjj2DdOvjuO2vjSHYk88Cc\nB1jcezGlb8v9V6AyO8im7LARxRtvmGQgC8iE1VJSoEYNU0m8WTOrozFdt8oOCxTcQLqDbCjZkUyT\nWU2IvBppaRxDh5p/dOfOWRqGEHz3Hdx1lz0SAJBnEoCrJAl4SD6/fHz/+PeUuc3a3ZzuvttsKiWD\ncMJKWsP48abcuZVOXjpJx/kdfbZEhDPSHeQDjhyBJk1MSYlixayORrjVuXOwZw+0aWN1JBlatsx0\nTVpdIkJrzd5ze6lTxr77K+SEdAfZ3CdbP2Hvub2WtX/PPWb90KefWhZC1q1aBWFhVkeRe0yfbvst\n5bSGd94xScDqHhilVJ5LAK6SJOAFFYpVoHA+a0tNjxhh1mTFxVkaRub+/BPefdfqKHKHuDgzJfTl\nl62OJEPr10NkpOmWtEp4dDjfhH1jXQA2JknACx6p+QhVS1a1NIa6dc1Ws59/bmkYmZN9iLMul+wf\n/O67plqolXWsLidcJi7J7t+ArCFjAl4UFRfFhdgL1LizhiXtb9oEvXubxbn581sSQta89BLkyyf7\nEGfE4YDatU0fX1CQ1dGka8cO6NrVjEsVtEe19TxJxgRyiZWHV7L44GLL2m/a1BSYtP32k8OGmc2S\nL12yOhL7Wr4cbrsNWre2OpIMjR9vKltblQCOxxyXK4BMyJWAj/nlF/MZu2cP+Nn5K0CvXvDAA/B8\n7tnn1auuXoWTJ6FmTasjSdf+/WZfi2PHTL6ywqhfR1GvXD0er/O4NQF4iawYzoUSUxIp4O/9Orpa\nm53H3ngDunf3evNZd+6c2S/TDrWGRY489RRUqwZvvWVdDNc+T/L6wjDLu4OUUp2UUvuVUgeVUq87\n+f1LSqkwpVSoUuoXpVRFd7SbWx2OOkzQF0FYkfCUgjffhHHjTEKwrdKlJQHkYkeOmHLRVl3IXVsM\nppTK8wnAVS4nAaWUHzAN6AjUAXorpW6errADaKC1rg98D/j0iF/1O6qztM9Sy96cXbuarqAff7Sk\neeED3nkHnnsOSpb0ftsO7aDZ7Gb8dfkv7zeeC7ncHaSUagqM1lo/lHp/BKC11hPTOb4+MFVr3dLJ\n73ymO8hqixfDqFFmBaetxwZErnP4sJmEcPiwdYVhT106Rfli5a1p3AJWdweVB66f1H0y9bH0DAKW\nu6HdXE9rzVM/PcWJGO/Pie/SxczYWLTI602LnLp0Cb6x/4Kn//3PFC60sjK4LyUAV+VzwzmcZR+n\nX+eVUn2BBkC689rGjBmT9nNQUBBBNp4D7SqlFAPrDaRs0bIWtA1jx8Jrr5kBYltfDSxYYEaza1iz\nvsI2Zs2CrVvhcfvOdDl0CJYuNVcB3nbs4jE+3voxkzrk/d7mkJAQQkJC3HIud3UHjdFad0q977Q7\nSCnVDpgCtNJaX0jnXHrDhg00s0ut2TxOa1PW96WX4IknrI4mA2+/bWYL2XzXLI9KSoLq1eH776Fh\nQ6ujSVf//iZXWzEj6FLCJTac2ECn6p2837hFUlJS2Lx5My1atMhxdxBaa5dugD9wGKgMFABCgftu\nOiYg9Zh7MjmXrlChgn700Uf16dOntS/ZcHyD3nJyi9fbXbFC61q1tE5O9nrTWXfmjNYlSmgdGWl1\nJNZZuFDrVq2sjiJD+/drXaqU1tHRVkeS9zkcDv3zzz/rOnXq6DZt2mjzUZ6zz3CXOwG01inA88Aq\nIAwI1lrvU0qNVUp1ST3sPeA24Ful1E6lVLrzUg4ePMiDDz5IMR+reXwu9hwX4y96vd0OHeCOO+Dr\nr73edNZd24fYV68EtDabRb/yitWRZGjcOLMQsXhx77a74cQG9p/f791GbWDp0qVMmDCBX3/91aXz\nyGIxwerVZjpfWJgp2WNL+/aZmvnh4VCokNXReNfvv8OgQeY1sOngzZ490LatGRPw9ve3+bvmU65o\nOdpVa+fdhm3E6tlBXjFt2jR+9IGJ7Q7tYPtf273aZtu25sv2vHlebTZ77rsPGjTwzcUNzZrBypW2\nTQBgphuPGGHNpkV96/bN0wng/PnzfOrBzUDs+666SYcOHahXr57VYXjc6cunGbt2rFe3v1MKJkyA\n0aMhPt5rzWbf/Pk2H8H2EH9/qFLF6ijS9ccfZr3JM894t9195/Z5t0GLFC5cmKioKI9VGMj13UFa\na1kW7ibdukHLlrbvehY2orUpEjdoEAwc6L12E5IT6DC/Az8+8SMlC1uwLNlmfKI7yJnff/+dRo0a\nsXr1aqtD8YiY+Biv1hd6912YOBGio73WpMjlli+HqCjo18+77RbMV5CQASF5KgEkJiYyffp0ZsyY\n4dV2c3USaN68Oa+++irPPPMMH3zwgdXhuN3TPz/N+uPrvdZe7drwyCPw3ntea1LkYg6HqUb7zjve\n2zUs2ZGctj9AXuoBOHToEPfddx+LFy+mUaNGXm0713cHASQlJREbG0txb89N87CE5AQK5vPubhwn\nTkD9+rB7N9x9t1ebFteLi4PZs820LZt+2C1cCFOnwoYN3gsxeE8wfxz/g6mdp3qnQS9JSkpi48aN\ntGrVKkfPl/0EnHA4HJw/f54yZcq45XxWc2gHfso7F26vvQYxMfDZZ15pLmfef98MElfMo1XJZ8yA\nn3+GJUusjsSphAQzYWvOHO/ubqm1Ji45jiL5i3ivUQ9wOBz4uXG2l8+OCWRkz5499O3b1+ow3GLz\nyc30+KaH19obMcIUlttv5/U3p07BRx9ZHYVnOBzwwQdmX0abmjoV/vEP7yWAa18OlVK5PgF88skn\nvPPOO1aHkSbPXgmAqavh763OSg9yaAdnrpzh7tu91z8zeTL89pttv4iaRWMNGpi9C/Pa6vIlS8x8\n3W3bbNkVdO6cGT/6/Xfv7G6ptab9vPZ82uVTqt9R3fMNelhUVBSFChWiSBH3JTPpDsqGDRs20KRJ\nkzyRHDwpMRHq1IFp06BjR6ujSUevXtCkiamAl5e0aQODB0OfPlZH4tRzz5mV5VOmeK/NYxePUaVE\nlVw3GBwdHU3x4sU9Hrd0B2VRSkoKo0aNom7duvz000+WbO/oqqd+eoo9kXs83k6BAqbb/eWXITnZ\n483lzCuvwIcf2jjAHAgLM3sz9uxpdSRO7d0L335rCrt6U9WSVXNVAoiNjWXixInUqFGDnTt3Wh1O\nhnwqCfj7+7NmzRomTpzIypUrrQ4nR4Y2HkrNO71wDY7ZhrJcOTNGaUuNGpmVtG6qq24LtWvDli2Q\nP7/VkTg1fDiMHAl33un5tn458gujfh3l+YY84L333mPbtm38/vvvBAYGWh1OhnyuO0hkz65d0L69\nGSS2Yr/YTMXGghv7VkX6Vq40O4bt2WOuFD3tcsJljsccp06ZOp5vzM3cPfsnM9Id5CbTp0/n/Pnz\nVoeRJXvP7WXE6hEeb6duXXj0UbNloC1JAvCKpCTT+zZpkncSAMDtBW/PFQlg/fr1JN/UJenNBOCq\n3BOph2mtuXTpkltH7D2pcvHKXqucOG4czJ0LBw54pTlhQ9Onm8WDXbt6th2HdjB02VD+uvyXZxty\no88//5yIiAirw8gx6Q4SWTJ5sukOWLnSlrMWhQedPg3332+qhXpjSuiP+3/k4RoPk9/fnuMidiTd\nQR40ffp0JkyYQGxsrNWhpGvKpimEngn1aBsvvGA+DL77zqPN+KakJHO5ZdNZTq++amaseiMBAHSr\n1c2WCSA8PJxP8uDudpIEMtGuXTt27NhB9erV2bt3r9XhOHXvnfdSukhpj7aRPz98/LGZMnr5skeb\nyrnhw81Kptzmu+/M9m423NZt7VpYtw7efNOz7Sw+sJhvw771bCM55HA4eOGFF2jQoAFnzpzJlVPL\nMyLdQVkUGhpKnTp1yG/TqXveMnAglC5tBghtZ/BgU0vI25PYXaE1NGxoVgh7usM9m5KSICAAxowx\nWzx70p7IPcQnx9Pw7oaebSiHZs+ezSOPPGLbWmSyYtgiCQkJFChQwDaLWC4lXGLSH5MYHTSafH6e\n+VYZGWlqxqxZY/5rK7lxH+KQEBgyxJb7B//f/5n9AmQcyP5kTMAi06dP591337U6jDSF8xWmYvGK\nHq02WqYMjB1rSgfYLl9f24fY1psl32TyZNPHZrMEcOKE2Sdg6lTPJYDo+GhGrB5BssM+YyFhYWEM\nGDDA6jC8Sq4EXOBwOIiNjaVo0aJWh+JVKSnQtCk8+yw89ZTV0dxkzRqTocLCbPfBeotTp0xX0NGj\nULiw1dGk0dr0TDVq5NmetfjkeL7e8zX96/W3zdV0YmIiYWFhBAQEWB1Ktkh3kI04HA4OHTpETW9N\npUjHhhMbOHjhIAPrD/TI+UNDoUMHs6K4XDmPNJEzWpsM9emnpkPb7mJiwGabIX39tZmstGOH9xaG\nWUFrTVJSEgXywB8p3UE2cvToUVq1akW/fv04evSoZXGULlKaisU8t+FK/fpmHPb55z3WRM4oZSa0\n54YEALZLABcuwIsvwsyZnksAo38bzeGow545eRatXr2aJk2aMH36dEvjsANJAm5WvXp1Dh06RPXq\n1Vm4cKFlcdS4swZtq7X1aBtvvWXqyHz/vUebyT4bTrXMLYYPNwVMmzXzXBsBdwVQ9raynmsgE6tX\nr+aZZ57h5ZdfZtiwYZbFYRfSHeQD3l3/Lp1rdKZ+ufpuP/cff5gPjbAwmxaYE1m2ejUMGmQS++23\nWx2N5zgcDlJSUvLUdG/pDsolkpKSmDJlCikpKV5tt3H5xlQoVsEj527RAh57zExwEbnXlStmpuqn\nn3omAczfNZ95f3p/1tahQ4c4c+bMDY/5+fnlqQTgKkkCXnT58mWuXLni9V3N2lVrR6kipTx2/nff\nNVtRrljhsSbyDocDXn/dlMC2keHDoWVLeOghz5y/aYWmNKnQxDMnz0BwcDChoZ4tqZLraa1dvgGd\ngP3AQeB1J79vCWwHkoDuGZxHC89JSE7Q3YK76fNXz7v93KtXa12+vNbn3X/qnEtJ0bp/f61jYqyO\n5G+LF2sdEKC1w2F1JGmWLdO6cmWto6OtjkTkVOpnZ44+v12+ElBK+QHTgI5AHaC3UqrWTYdFAAOA\nBa62lxcNGzaMzz77jMTERI+2U8C/AMObDeeOwne4/dxt25qxgf/8x0aLyPz8ICEBZs+2OpK/TZ5s\nvnbbZF78hQvwr3/BF1+4f6LStr+2MfDHge49aTrOnj3LuHHjbqnrLzLnju6gxsAhrXWE1joJCAYe\nvf4ArfVxrfUewC4fD7bSp08fFi1aRK1atbh06ZJH22pRqUXawhyHdrj13OPHmz1o589362ldY6d9\niLdvt9X+wVrDM8/AE09AUJD7z1+vbD1ea/Ga+098k/Hjx1O7dm3OnTtHfHy8x9vLa9yRBMoDJ667\nfzL1MZFFTZo0YeXKlSxZsoRixYp5pc2I6Ahaf9HarYmgUCFYsMAMEttmj41GjaByZXvUwJ482dTk\ntsmgZHCwmQn0zjvuPW98svkgzu+fn9qla7v35E7Uq1eP0NBQPvroI59bve8O7phQ7ey6Nsff+MeM\nGZP2c1BQEEGe+IpiU7Vr3/oPJioqihIlSrh9u7rKJSqzsPtCt9cZql/f9Hb0728qOHh5DNy54cPN\nEtgnnrCuGyYmxhSLs0k9+ogIGDbMFIhzZ8WKs1fO0nF+R7b9e5tHihhqrW8pMdG5c2e3t2N3ISEh\nhISEuOVcLq8TUEo1BcZorTul3h+BGaSY6OTYz4HFWutF6ZxLuxpPXjN06FBatGhBr169PNaG1pqo\nuCjuLHKnW86XkgIPPggdO8LIkW45pWscDmjSBL79FqpUsS6O+HhbVDdNSoJWrczU3uHD3X/+6Pho\nShQq4fbzXrhwgY4dO7Jp0ybyyYLAG1haO0gp5Q8cANoCp4EtQG+t9T4nx34OLNFaO11jKkngVlpr\nHA6HR6eVrotYx7Qt0/im5zduO+fJk6Y22jffmA8cy2ltm8FYq73+uukGWrzYfTX2Dkcdpvod1d1z\nsgxERETiLVEzAAAgAElEQVRQuXJlj7eT21heQE4p1QmYghljmK21nqCUGgts1VovUUo1BH4ASgDx\nwBmt9f1OziNJIAuioqLYvXs3rVu3dts5E1MSKeDv3mIxy5eb+kI7dpgS1MJ6y5aZRWE7d0IpNy0d\nSUhO4MEvH2RJ7yWULOyeZeOJiYlcuHCBu+66yy3ny+ssTwLuIkkga3bs2MHjjz9O5cqVmTBhAo0a\nNXLbuSOvRnLs4jG3LewZOdJMilm+3P6VnfO6a1dn335rFoa5k7O++pxITEzkiy++4N1336V///78\n97//dUN0eZ+UjfAxgYGB7Nu3jz59+nDOzXvq7j+/n7URa912vv/+13SF22jvHZ+UnAxPPglDh7on\nAWit+XTbp8QlxQG4bT+A8PBwfvjhB7766itJAF4iVwLC4/76y2z4tWCBGTD2GVqbusxvveW+vpcc\neukls4Pl0qXumbHl0A7eWfcOzzd+3m1dQCLn5EpApDl16hT//Oc/cUcy/eXIL0zf4nq99bvvhoUL\noU8fOHbM5dO5Jj4eHn0U4uI839a6daag0h3uX6GdHfPmmUHgr75yPQFce1/5KT/eav2WSwng+++/\nZ926da4FJFwmSSCPKVu2LKNGjXLL5fl9pe+jUXn3jDe0aQOjRkG3bqZipWUKFTJ9I95Y1myD/YO3\nbTMh/Pij66W+UxwpdFrQiVOXTrkltjvuuMNriyNF+qQ7yEdERkZSunTpHCeHFEcKsUmx3F4w53WG\ntTZ1amJizNRRyz4bvbEP8f790Lo1hIdbtn9wZKRZMP3BB2ZNgDuERYZRu3TtbL+P3DVwLJyT7iCR\nqUGDBvHAAw/wyy+/5Kir6If9P/D66tddikEp+PhjM0YwbpxLp3JNmzbmg3n5cs+18X//Z6rpWZQA\n4uJMr1f//q4ngH3n/l7yU6dMnWx9mMfHx/PJJ59Qq1YtLly44FogwjNyWn7UEzeklLTHJCcn6wUL\nFuimTZvqS5cuZfv5DodDxyXFuSWW06dN6eJ589xyupyZP1/rNm08c+6EBK2rV9f67FnPnD8TKSla\n9+ihde/erlesTkpJ0kFfBOkzl8/k6Pldu3bVnTt31hs3bnQtEJEhXCglLd1BIttOxJxgxeEVDG4w\nOMfn2LvXfCH/6iuLZgwlJZnumiVLPDNwm5JiWeGk116DjRvNdpEFC7p+Pu1CV05sbCxFihRxPQiR\nIekOEi4JDg7mt99+y/LxyY5kl1cX165txgV69YLdu106Vc7kzw8bNnhu5o5FCeDTT80g8I8/5jwB\nRMdH8+SiJ0lITgCytgYgJiaG75xUapUEYH+SBAQVKlSgdOnSWT6+asmqDKg/IO3+pYSc7YHQujVM\nmQIPPwzHj+foFOI6335rFuctWwZ3ulALsHjB4gysNzBbiV5rzYYNG9wyNVl4l3QHiXTFxcVROJOB\nzYjoCB775jG2DN6S47LUH35oBozXrYNy5XJ0Cp+3YgUMGACrVkG9etl/flJKErvO7qLB3Q3cH5zw\nOOkOEm539OhRKlWqxJgxYzKc1VG5RGXWP7XepX0JXnwR+vWD9u3Ndocie37/3bx+P/yQswQAcOTi\nET7a8lGmx+3Zs4cBAwYwc+bMnDUkbEeSgHCqWrVqbNiwgVOnTjFq1KgMjy2c31wtJDuSGbJ4COdj\nz2e7vTffhM6doVMns44gV3ruObMuwIu2bIHu3U1JjubNs//8azvL1SpVi7nd5mZ47A8//ED79u2p\nWbMmPW2yRaZwnXQHiUxldXaI1pqfDvxE15pdc3RloLUpcLZtm5nC7+oK1yy7eNHs+7tiBeR0s5Lt\n2+Gf/zR7CHtp+8iNG81agDlzoEuX7D//yz+/ZN+5fYxvNz5Lx1/bv7eQDTbGETeS7iDhUTcnAK01\nw4YNIzIy8pbjutXqlpYA1kes50TMCbJKKZg6FZo1M9NG3VwgNX0lS0JiInzvdK+jrPHy/sHr15sE\n8OWXOUsAAN1qdeONlm84/d2sWbO4fPnyDY8VKlRIEkAeJElAZJvWmubNm3NnJlNQQs+Ecjwme9N+\nlDJlDjp3hqAgOH3ahUCz45VX4P33zeVIdh0/bq4iBud83UR2/PqrWQW8cKHpPsuOkb+OZP/5/QAU\nK1iMYgWd1+6JiYnh4sWLroYqcgHpDhJuc+HCBfz9/SlR4tb9ZROSEzh95TRVSlTJ8vneeQe++MJ0\nDVX39M6FDgfUqgWzZmV/P8zhw83zP/jAM7Fd56uvzED6t9/mbNvO1UdX0/DuhjfsARwfHy/f8HM5\n6Q4StrB8+XKqVq3Ks88+y+HDh2/43ba/tjF27dhsnW/UKLP6tWVLs67Lo/z8TLnNyZOz9zytzdzW\nYcM8E9d1zUyebPYH/vXXrCeAqLgopm6emna/XbV2lChUAofDwdKlS+nQoQO9e/f2UNQiN5AkINym\nb9++hIWFUapUKSIiIm74XYtKLZjTdU7a/cirkTc/3anBg83AZ7durnXZZ0n//qbOdXb2GlAKNm8G\nD25+npxsvv1//jn88Qf84x9Zf27hfIWJTYpNmwV0zdmzZxk3bhx9+/YlODjYzRGL3ES6g4TXXKsj\n49AOGs1sxM+9fqZ8sfJZeu7OndC1q0kKb77pO/sVnz9vSmv4+0NwcNZmTK0NX0uxgsUIuCvA8wEK\nW5DuIGF758+fp379+qSkpOCn/Nj8r81pCeBSwiWi4qIyfH5AgJkT/8svJhn4wphlaKjZD6BBA1MK\nIqtTZqPiooiOj067P2LECL799lsPRSlyO7kSEF6T3gDkon2LWBexjg87fZjpOZKSzDjs0qXw3XdQ\nv74nIrWW1mZ8euRImDYNnngi4+MvJ1zm460f81qL15yu54iIiKBs2bIy+JuHyZWAyBWcfQhNnz6d\n5R8sp88dfdKKj4WEh3A18arTc+TPb4rOjRtnykxMnGiqNucVFy6Y6Z/Tp8PatZknAIAi+YuQ4khh\n0U+LGD/+1oVflStXlgQg0iVJQFiqe/fuVKtWjd69e/PDDz8AMO/PeTd0ZzjTu7dZWbxsmVlP4JEN\n7DO6Kn3mGdi1y63NrVxprmyqVTNjzbVrp3/s+PXjWXF4BQCXYi7x0eMf8cGkD6hUqZJbYxI+IKe7\n0Xjihuws5rNSUlJ0UlLSLY8fPX9Uv73m7Qyep/X772t9551av/ee1omJbgooIkLrli1NAzfbt0/r\nMmW0jo11S1OnT2vdq5fWVatq/csvzo9xOBz63NVzaff/PPOnPn/1/HXhRrglFpE74cLOYnIlIGzB\nz8+PfDfV7YmNjaVVi1bcX/r+tMeuJF65oWa9n59Z7Ltpk9k/vn59CAlxQ0AVK5rpos72IXbT/sHJ\nyaaEdt26ULUq7NkD7do5PzYkPIRnlz7LSy+9xJo1a6hbti53Fvl7xbZcAYickoFhYWtXrlyhaNGi\naffHhoylsF9hhjYaesteB1qbHbVefNHMJvrvf80HbI4tWACzZ5vscs25c3DvvXDgAJQpk6PTam3W\nPIwaBRUqmDGOm+f+JyQn8MavbzCp/ST8/fzT5vkfOniIihUryo5d4gYyMCzyrOsTAMDbrd/m3ph7\nKVeuHI899hhPznqSw1FmdbJSppDngQNmnKBDBzPHPiwsh40//jgcOmQWKVzz8cem4mgOEkBKiklS\njRvD+PFm5s/q1X8ngJ2nd3Iw/CDvvfceHdt1ZOeqnSQ5kgDwU374KT9q1qwpCUC4lVuSgFKqk1Jq\nv1LqoFLqdSe/L6CUClZKHVJKbVRKybWryBGlFN0e6caRI0d45JFHqJK/CqWL/L015qwds9D+cbz4\nIhw+bDZZadvWzCT6+edsziTKn99UBr2+lMT27fDSS9mK+coV84Ffs6b58H/tNdi6FVo/mMjVpCs3\nxL7t6DbCw8N55ZVXWPa/ZRTKJ7N6hIfldDDh2g2TSA4DlYH8QChQ66ZjngE+Tv35CSA4nXN5ZNBE\n+IaE5AT94vIX9X+e/Y/+6aefdEJygp7/53wdH6/1vHlaN2qkdZUqWo8cqfXu3Vk8aXS01t27a52c\nnK1YkpK0XrVK6379tC5e3Jxi7fpEHRV7UWut9dmzZ3XV/1TVc3bMyeZfKcStcGFg2OUxAaVUU2C0\n1vqh1PsjUgOaeN0xK1KP2ayU8gfOaK1v2dlcxgSEO0RHR+Pn50e8fzyT/pjEpA6TAHhz4pusv7CL\nUkf+zebNrSla9HY6djTdRi1agJPip1mmtZmm+scfZtrqqlVQqupaajbYQvXbLvH++2OZumUqlxIu\n8Vbrt3A4HGzdupXGjRtnacMeITLiyphADrdRukF54PqdQ04CjdM7RmudopSKVkrdobXOuFaAEDlw\nrZR1MYqlJQCAihUrUihqK1euTGXNmlqsjwhl2qYZ7Hp/Hr16QcEyk7j3gUq0q/wE1arBHeUu41/0\nIkVTNP7+DuIdieQvehsFEytw/jzsPX6GncciCFu5h7Nne5Fydyjlmq7lxbYjmTwZHvzXEI468tGg\neE8SEhJ4ockLaR/4fn5+NGnSxJLXR4jruSMJOMs+N3+dv/kY5eQYAMaMGZP2c1BQEEFBQS6EJsTf\nhvQZwpA+Q9LuV65WkU5NqlH+LTNW0PflCByVNH6XTbnm3Ve2c6r4N1z5OoykpAh0lSsUbtyOOkeC\nKVUK/CtHkFhhNbVrJ/Hll/GUrlaVvy4XoFFqTbz9y/Zb9JeKvC4kJIQQt8yFdsMU0dTuoDFa606p\n9511By1PPeZad9BprfUt0yukO0gIIbLP6imiW4HqSqnKSqkCQC/g55uOWQwMSP25J7AGIYQQlnO5\nOyi1j/95YBUmqczWWu9TSo0FtmqtlwCzgXlKqUPABUyiEEIIYTFZMSyEh1WpUuWWndaEyKnKlSsT\nHh5+w2OudAdJEhDCw1L/gVodhsgjnL2frB4TEEIIkUtJEhBCCB8mSUAIIXyYJAEhfEzVqlXZu3fv\nDY9prenRowf33XcfAQEBdOzYkWPpbNc2d+5cevbs6Y1QhRdIEhBCADBw4ED27dvHzp076dq1K4MH\nD073WKl3lHdIEhBCoJSiS5cuafebNWvG8ePHs3UOh8PB8OHDuf/++6lbty6vvvpq2iyWGTNmULt2\nbQIDA6lfvz4HDx5Ea82zzz5L7dq1CQgIoGXLlm79m0TWuKN2kBAij5k2bRpdu3bN1nNmzJjBrl27\nCA0NRWtNp06dmDFjBkOGDOG1117jwIEDlC1blqSkJFJSUvjzzz8JCQlJ65qKiYnxxJ8iMiFXAkJY\nSCnXb+42adIk9u/fz//+979sPW/16tUMHDgQf39/8uXLx1NPPcXq1asBaNu2Lf3792fatGmcPHmS\nQoUKUa1aNZKTkxk0aBDz58+XtRQWkSQghIW0dv3mTtOmTSM4OJjly5dTqFD2djXTWt8yVnDt/vff\nf88777xDbGwsbdq0YeXKlRQrVow9e/bwxBNPsGvXLurUqUNkZKTb/haRNZIEhBCA6c6ZMWMGq1at\nonjx4hke6+xbe/v27fniiy9ITk4mKSmJuXPn0q5dOxwOB0ePHqVhw4a89tprdOjQgZ07d3LhwgVi\nY2Pp0KEDEyZMoESJEhw9etRTf55Ih4wJCOFjlFK0a9eOfPnypX1737BhA8888wxVqlShffv2aK0p\nVKgQGzdudHqO5cuXU6lSpbTnP/XUU4wZM4bDhw8TEBCAUopOnToxePBgkpKSGDhwIDExMSilqFSp\nEhMnTiQ8PJzBgweTkpJCcnIynTt3pmnTpl5+NYTUDhLCw6R2kHAnqR0khBDCbSQJCCGED5MkIIQQ\nPkySgBBC+DBJAkII4cMkCQghhA+TJCCEj3FWShpg8uTJ1KpVC39/f5YtW5bu89euXUujRo08GaLw\nIkkCQggAgoKCWLZsGa1bt870WCklnXdIEhBCANCgQQOqVavm0sK2iRMnppWSHjRoELGxsQD89NNP\n1K1bl8DAQOrWrcu6desAGDt2bFqJ6QYNGnDp0iW3/C0i66RshBDCLVasWMGCBQvYtGkTt912GwMG\nDGDcuHGMHz+e0aNHM3PmTJo0aYLWmqtXrxIdHc2HH37ImTNnKFiwIFevXqVw4cJW/xk+R64EhLDS\nmDHO60OPGZO149M7zgKrV6+mV69e3HbbbQD8+9//Tisl/eCDD/LSSy/x/vvvs3fvXooWLUqxYsWo\nUaMG/fv3Z9asWVy+fBk/P/lI8jZ5xYWw0pgxzutDZ5QEsnKcBTIqJf3BBx8wc+ZMChYsSM+ePZk9\nezZ+fn5s2rSJ559/npMnT9KgQQP27NljReg+TZKAECLb0islHRwczNWrV9FaM2vWLNq1awfAwYMH\nqVOnDkOHDqVv375s3bqVq1evEhkZScuWLRkzZgz/+Mc/JAlYQMYEhPAxzkpJ7969m5kzZzJlyhTO\nnz/PwIEDKVSoUFrXzc127959Qynpdu3aMWfOHHbt2kXTpk1RStGwYUPefPNNAEaMGMHhw4fx9/en\nZMmSzJ49m+joaB577DHi4+NJSUmhQYMGdO/e3dsvh8+TUtJCeJiUkhbuZKtS0kqpkkqpVUqpA0qp\nlUopp9sRKaWWK6UuKqV+dqU9IYQQ7uXqmMAIYLXWuiawBngjnePeA/q62JYQQgg3czUJPArMTf15\nLtDN2UFa69+AKy62JYQQws1cTQJltNZnAbTWZ4DSrockhBDCWzKdHaSU+gUoe/1DgAbe9FRQQggh\nvCPTJKC1bp/e75RSZ5VSZbXWZ5VS5YBIVwMac93il6CgIIKCglw9pRBC5CkhISGEhIS45VwuTRFV\nSk0EorTWE5VSrwMltdYj0jk2CHhFa/1IBueTKaIiz5EposKdbDVFFJgItFdKHQDaARNSA2qglJpx\nXYDrgK+BB5VSx5VS6V5dCCE8q0qVKtSuXZv69etTt25dvv76a6fHOds3ICwsjKpVq2baRkREBKVL\ne2aI0M/PL606qTfExcXRqFEj4uLiAIiMjKRjx47UrFmTgIAAtmzZ4vR5s2bNol69etSrV4/69euz\nYMGCtN+NHTuWsmXLEhgYSGBgIEOHDk37Xa9evdi0aZNn/6jruLRiWGsdhfnwv/nx7cC/r7vfypV2\nhBDuo5Ti+++/57777iM0NJTmzZvTvn177rjjDqfHZuWx9NpxlcPhuKWonLf3Mpg6dSo9evRIq3D6\nxhtv0Lp1a1auXMkff/zBk08+yaFDh2553r333su6desoXrw4p06don79+rRs2ZJKlSoBMGDAAN57\n771bnjdy5EheeOEFt3X3ZEZqBwnhg651J9SvX5/bb7+dY8eOZet5cOu3/Zvva60ZPnx42rfh33//\nPe13y5cv54EHHqBRo0a0aNGCzZs3A+bqo169ejz99NMEBgayYsWKDGO43ooVKwgMDKR+/fq0b9+e\nI0eOAKZuUfPmzQkICKBu3bp88MEHQPp7HNxsxowZ9OnTJ+3+N998w3/+8x8AWrRoQeHChdm+ffst\nz2vVqhXFi5v1s+XLl+euu+7i5MmTmf4ddevW5dy5c2nxe5zW2jY3E44QeYvd3tdVqlTRYWFhWmut\n16xZo4sXL65jYmJuOS4kJEQXKVJEBwQEpN1q1aqlq1atqrXWOjw8XJcuXTrt+Ovvh4eHa6WUnj9/\nvtZa67Vr1+oKFSroxMREfeTIEd2sWTN9+fJlrbXWYWFhulKlSmlt5suXT2/evDnd+JVS+urVqzc8\nFhkZqUuXLq3379+vtdZ69uzZukmTJlprrYcNG6YnTJiQdmx0dLTWWut69erpTZs2aa21djgcafFc\n78SJE/quu+5Ku3/hwgVdtGjRG47p3Lmz/uGHH9KNV2utf/vtN12pUiUdHx+vtdZ6zJgxumLFirpe\nvXq6Y8eOeuPGjTcc//TTT+vPPvvM6bmcvZ9SH8vR565cCQhhoTFjxqCUQil1w8y463+f3uPpPScr\nevToQWBgIGPHjmXRokUUK1bM6XF16tRhx44dabfvvvsuy20ULFiQJ598EjDfiosUKcKBAwdYuXIl\nR48epVWrVgQEBPDkk0/icDg4d+4cADVq1KBx48bZ+ns2b95M/fr1qVmzJgBPPfUUoaGhXL16lVat\nWjFr1izefvttfvvtt7Rv523btr1lj4ObnTx5krJly97yeHbs3buXAQMGEBwcTMGCBQF45plnOHbs\nGKGhoQwfPpxHH32Uixcvpj2nXLlyN1w1eJJUERXCQul9yF//+5w8LzPXxgSu1717d44dO4ZSivXr\n12d6jnz58uFwONLux8fHZ3i8w+FIm9nSqVMnvvjiC6fHOfswvp6zMQGdwV4G3bt3p3nz5qxatYoJ\nEyYwZ84c5s2bx+TJkwkLC2PNmjX07NmTV155hUGDBt1wjsKFC9/wd10bN4mKikr7+fjx41SsWNFp\nrIcOHeLhhx9m5syZNGvWLO3xMmXKpP3crl07KlasyJ49e2jZsiVgXstSpUpl+Dq4i1wJCOGDtJP+\n6EWLFrFz50527NiRtjtYRs8tV64cSUlJHD16FOCG2S8ACQkJLFy4EID169eTkJBAzZo16dChAytW\nrGDv3r1px27bts2l2Js1a0ZoaCgHDx4E4IsvviAgIIDbbruNI0eOULZsWfr378/o0aPZunUr4HyP\ng5vVrFmT06dPk5SUlPZYz549+eSTTwD4/fffiY+Pp0GDBrc89+jRo3Tq1ImpU6fSoUOHG373119/\npf0cGhpKRERE2lUMwL59+6hXr16WXxNXyJWAED7G1dk1157v7+/PlClTaNeuHWXKlOHhhx++4bhS\npUoRGhrKxIkTAQgODiZfvnxUr16d+fPnM2jQIOLj40lMTKRFixY0bNgwy+3XrFkz7aqiaNGi7Nu3\njy+//JLevXuTkpJC6dKlmT9/PmAGchcsWECBAgXw8/Pjo48+ApzvcXCzQoUK0aZNG0JCQmjf3sxs\nHz9+PH379mXu3LkUKVIkrR2AwYMH8+ijj9KlSxdGjBhBVFQUb7/9Nm+99RZKKSZOnEj79u0ZOXIk\nO3bswM/Pj4IFCzJ//vy0q4PY2Fj27t3Lgw8+mJ3/LTkm+wkI4WGyWCx327hxI5MmTWLRokVeaW/G\njBmcOnWKsWPHOv293RaLCSFEntasWTO6dOmStljM0/Lly8eIEU4LL3iEXAkI4WFyJSDcSa4EhBBC\nuI0kASGE8GGSBIQQwodJEhBCCB8mSUAIIXyYJAEhfIzsJ5A9N+8nEBQUxD333ENAQACBgYHMnTvX\n6fMcDgfPPfcc1atX5957771hMdrrr79OcHCwV+LPjKwYFsLHyH4C2XPzfgJKKaZNm8ZDDz2U4fMW\nLFjA0aNHOXz4MOfPnycgIID27dtTqVIlXn31VR544AF69erljT8hQ3IlIIQPujbPXPYTyP5+AsAN\nhfPS8/XXXzN48GDAlNDo1q0b3377bdr9e+65h19//TXT83hcTmtQe+KGzequC+EOGb2vR/82Wo/+\nbXSO7+eE7CeQ8/0EtNY6KChI165dW9etW1f369dPnzp1ymmc999/v962bVva/ffee08PGzYs7f5/\n//tf/cYbb6T7d6bH2fsJF/YTkO4gISw0JmiMS/dzqkePHhQqVIhixYplup/A9XvohoWF8cgjj2Sp\njfT2E1i/fn3afgI69Vu9J/YTePbZZ9P2E3j99de5evUqbdq0oU2bNsDf+wl0796dhx56iDp16txy\nXmf7CcyfP5/y5cujtebdd9/liSeeyFLp7ZuVK1cuR89zN+kOEsIHff/99+zYsYOQkJC0apXdu3dP\nG+y8evVqpudwdT+BHTt2sHPnTnbu3MmJEyfSupI8sZ/A+vXrqV69OhMmTKBfv34ATJ48mZkzZ1Kw\nYEF69uzptIrozfsJgNkq8tr5hw0bltaVdbNKlSoRERGRdv/mfQfi4+PTxhmsJElACB907Rv49WQ/\ngcz3E0hJSSEyMjLt9wsXLuT+++93GmfPnj2ZOXMmWmvOnTvHTz/9xGOPPZb2e2/uGZAR6Q4SwsfI\nfgI5308gISGBhx9+mKSkJLTWlC9f/oapng8//DDjxo0jMDCQfv36sXnzZmrUqIFSitGjR1OlSpW0\nY3/99VdGjRqV9RfeQ6SKqBAeJlVEczdP7CewatUqFixYkO4ag4xIFVEhhPAiT+wncPny5bQrJKvJ\nlYAQHiZXAsKd5EpACCGE20gSEEIIHyazg4TwsMqVK3u93o3IuypXruzW87k0JqCUKgl8DVQGwoHH\ntdYxNx1TD/gEuB1IAd7VWn+TzvlkTEAIIbLJyjGBEcBqrXVNYA3whpNjrgL9tNb3Aw8BHyqlnK9R\nF24VEhJidQh5irye7iWvpz24mgQeBa5NdJ0LdLv5AK31Ya31kdSfTwORgGcKjYsbyD8y95LX073k\n9bQHV5NAGa31WQCt9Rky+XBXSjUG8l9LCkIIIayV6cCwUuoX4PoyegrQwJvZaUgpdRfwJdAvO88T\nQgjhOa4ODO8DgrTWZ5VS5YDftNb3OTnudiAEeEdrne7aa6WUjAoLIUQO5HRg2NUpoj8DA4GJwADg\np5sPUErlB34E5maUACDnf4QQQoiccfVK4A7gG6AicBzoqbWOVko1AIZorf+tlHoSmAOE8XdX0kCt\n9S6XoxdCCOESW9UOEkII4V2Wlo1QSvVQSu1RSqUopQIzOK6TUmq/UuqgUup1b8aYmyilSiqlViml\nDiilViqliqdzXIpSaodSaqdS6kdvx2l3mb3flFIFlFLBSqlDSqmNSqlKVsSZG2ThtRyglIpMfT/u\nUEo9bUWcuYFSarZS6qxSKt1eFKXUR6nvy1ClVP2snNfq2kG7gX8Ca9M7QCnlB0wDOgJ1gN5KqVre\nCS/XycriPYCrWutArXWA1vqWtR2+LIvvt0FAlNa6BvAh8J53o8wdsvFvNzj1/RiotZ7j1SBzl88x\nr6VTSqmHgHtS35dDgE+zclJLk4DW+oDW+hBmrCA9jYFDWusIrXUSEIxZpCZulenivVQyAJ++rLzf\nrpzvtRwAAAIvSURBVH+dvwPaejG+3CSr/3bl/ZgFWuvfgYsZHPIoZho+WuvNQHGlVNkMjgesvxLI\nivLAievun0x9TNwqq4v3CiqltiilNiilJKHeKCvvt7RjtNYpQHTqJAlxo6z+2+2e2n3xjVKqgndC\ny5Nufr1PkYXPSo9XEc1gsdkorfXirJzCyWM+O5rtpsV7lbTWZ5RSVYE1SqldWutj7owzF8vK++3m\nY5STY0TWXsufgYVa6ySl1BDMFZZcWeVMjj4rPZ4EtNbtXTzFSeD6gbcKwF8unjPXyuj1TB00Knvd\n4r3IdM5xJvW/x5RSIUAAIEnAyMr77QRmWvRfSil/oJjWOqPLdF+V6Wt50+s2E7PmSOTMScz78pos\nfVbaqTsovX7BrUB1pVRlpVQBoBfm24O41bXFe5D+4r0Sqa8jSqlSQHNgr7cCzAWy8n5bjHl9AXpi\nBuHFrTJ9LVO/rFzzKPJezIwi/c/Kn4H+AEqppkD0te7hDGmtLbthBi5PAHHAaWB56uN3AUuuO64T\ncAA4BIywMmY734A7gNWpr9UvQInUxxsAM1J/bgbsAnYCf2IW7lkeu51uzt5vwFigS+rPBTGLJA8B\nm4AqVsds11sWXst3gT2p78dfgXutjtmuN2Ah5pt9AmZx7lOYWUD/vu6YacDh1H/bgVk5rywWE0II\nH2an7iAhhBBeJklACCF8mCQBIYTwYZIEhBDCh0kSEEIIHyZJQAghfJgkASGE8GGSBIQQwof9P4aP\nKredAALuAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fb9e59569b0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x_array = sess.run(x_vals)\n",
    "plt.plot(x_array, l2_y_out, 'b-', label='L2 Loss')\n",
    "plt.plot(x_array, l1_y_out, 'r--', label='L1 Loss')\n",
    "plt.plot(x_array, phuber1_y_out, 'k-.', label='P-Huber Loss (0.25)')\n",
    "plt.plot(x_array, phuber2_y_out, 'g:', label='P-Huber Loss (5.0)')\n",
    "plt.ylim(-0.2, 0.4)\n",
    "plt.legend(loc='lower right', prop={'size': 11})\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Categorical Predictions\n",
    "\n",
    "-------------------------------\n",
    "\n",
    "We now consider categorical loss functions.  Here, the predictions will be around the target of 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Various predicted X values\n",
    "x_vals = tf.linspace(-3., 5., 500)\n",
    "\n",
    "# Target of 1.0\n",
    "target = tf.constant(1.)\n",
    "targets = tf.fill([500,], 1.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Hinge Loss\n",
    "\n",
    "The hinge loss is useful for categorical predictions.  Here is is the `max(0, 1-(pred*actual))`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Hinge loss\n",
    "# Use for predicting binary (-1, 1) classes\n",
    "# L = max(0, 1 - (pred * actual))\n",
    "hinge_y_vals = tf.maximum(0., 1. - tf.multiply(target, x_vals))\n",
    "hinge_y_out = sess.run(hinge_y_vals)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Cross Entropy Loss\n",
    "\n",
    "The cross entropy loss is a very popular way to measure the loss between categorical targets and output model logits.  You can read about the details more here: https://en.wikipedia.org/wiki/Cross_entropy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Cross entropy loss\n",
    "# L = -actual * (log(pred)) - (1-actual)(log(1-pred))\n",
    "xentropy_y_vals = - tf.multiply(target, tf.log(x_vals)) - tf.multiply((1. - target), tf.log(1. - x_vals))\n",
    "xentropy_y_out = sess.run(xentropy_y_vals)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Sigmoid Entropy Loss\n",
    "\n",
    "TensorFlow also has a sigmoid-entropy loss function.  This is very similar to the above cross-entropy function except that we take the sigmoid of the predictions in the function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# L = -actual * (log(sigmoid(pred))) - (1-actual)(log(1-sigmoid(pred)))\n",
    "# or\n",
    "# L = max(actual, 0) - actual * pred + log(1 + exp(-abs(actual)))\n",
    "x_val_input = tf.expand_dims(x_vals, 1)\n",
    "target_input = tf.expand_dims(targets, 1)\n",
    "xentropy_sigmoid_y_vals = tf.nn.softmax_cross_entropy_with_logits(logits=x_val_input, labels=target_input)\n",
    "xentropy_sigmoid_y_out = sess.run(xentropy_sigmoid_y_vals)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Weighted (Softmax) Cross Entropy Loss\n",
    "\n",
    "Tensorflow also has a similar function to the `sigmoid cross entropy` loss function above, but we take the softmax of the actuals and weight the predicted output instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Weighted (softmax) cross entropy loss\n",
    "# L = -actual * (log(pred)) * weights - (1-actual)(log(1-pred))\n",
    "# or\n",
    "# L = (1 - pred) * actual + (1 + (weights - 1) * pred) * log(1 + exp(-actual))\n",
    "weight = tf.constant(0.5)\n",
    "xentropy_weighted_y_vals = tf.nn.weighted_cross_entropy_with_logits(x_vals, targets, weight)\n",
    "xentropy_weighted_y_out = sess.run(xentropy_weighted_y_vals)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Plot the Categorical Losses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEACAYAAACwB81wAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd4VNXW+PHvCk0gNEVAqUovqYg0kdA7KIqCemkqKoro\nVcDXn5Bwr91XrgqicEVA5F4sqIgKIiVcAUWkKb0JylW6QEgoKev3xwnzhmQmmZBJZpJZn+eZxzlz\n9tlnzUj27Nlnn7VFVTHGGFP0hfg7AGOMMQXDGnxjjAkS1uAbY0yQsAbfGGOChDX4xhgTJKzBN8aY\nIOGTBl9ESonIWhHZKCI/i0ismzIlRWSeiOwWke9EpJYvzm2MMcY7PmnwVfU80EFVo4BIoIeI3Jip\n2L3ACVWtD7wGvOyLcxtjjPGOz4Z0VDUp/WkpoDiQ+Y6ufsDs9OcfA518dW5jjDE581mDLyIhIrIR\nOAR8o6rrMhWpDvwGoKqpwEkRudJX5zfGGJM9X/bw09KHdGoALUWkSaYi4mbb8joYY0wBKe7rClX1\ntIjEA92BbRl2/QbUBH4XkWJAeVX9M/PxImJfAsYYk0uqmrlTnYWvZulUFpEK6c9LA52BHZmKLQSG\npD8fACz3VJ+qBvRjwoRYBg9W+vdXUlN9W3daWprP6oqNjfX7Z+V1nBs3ohERfo+lyHyeARCHxVlw\nD2/5akjnGmCFiGwC1gJfq+pXIjJRRHqnl5kBVBaR3cBjwFM+OneBE4Hp0+HIEXjmGV/X7XxJqyrJ\nqcm+rTyQlS0LZ874OwpjijSfDOmo6s9AtJvXYzM8Pw/c4YvzBYJSpeDTT6FlS2jYEIYMyfmY3Hhp\n9UuULFaSv7b+q28rDlShodbgG5PPfD6GHwxiYmIAqFwZvvgC2reH66+Hdu18d46RLUZSunjpPNVx\nMc5AFxMT4/Twz53zdyjZKlSfZyFgcRY8yc34T0EQEQ20mHKyZAkMHgyrV0Pduv6OppC6+P9ccrzu\nZIzJRETQgrpoG+y6doXYWOjdG06e9G3d+0/uZ+hnQ3N1YaZQErHG3ph8Zj18Hxo9GrZvhy+/hBIl\nfFNnaloqq35dRfs67X1ToTGmyPG2h28Nvg+lpEDfvlC7Nkydah1WY0zBsCEdPyheHObNg2+/hcmT\nfV//W+ve4uQ5H48ZGWOChs3S8bHy5Z2ZO23aQL160LOn7+pOSUsh8UIiFa+o6LtKA0lKivOzqFgx\nf0diTJFkQzr5ZM0auOUWWLYMwsL8HU0h0a0b/PWvzn+NMV6zIR0/a9MGXnsN+vSBw4d9W/fxpOPs\nPbHXt5UGgvLl4dQpf0dhTJFlDX4+uusu5w7cW27x7T1Fi/Ys4otdX/iuwkBRoYLv57UaY1xsSCef\nqcKgQRASAnPn2sydbD3xBFSrBmPG+DsSYwoVG9IJECIwcybs3Qt//7vv6084n+D7Sv2lQgUb0jEm\nH1mDXwBKl4YFC2DGDGfapq8cSTxC23fbkpKW4rtK/alSJTh71t9RGFNk2ZBOAfrpJ+jUCRYuhFat\nfFNnUnISZUqU8U1lxphCyYZ0AlB4uDO8078/HDjgmzqtsTfGeMsa/ALWu7dzTbJPH0jw4fD7fZ/f\nx9qDa31XoTGmyMnzkI6I1ADeA6oBqcA/VfWNTGXaAwuAfekvfaKqz3qor8gO6VykCg8+CP/9rzO2\n74sbS7cd3Ub9K+tTopiPsrYZYwqNAkueJiLVgGqquklEQoH1QD9V3ZGhTHvgCVXt60V9Rb7BB0hO\nhu7dISICJk3ydzTGmMKswMbwVfWQqm5Kf34G2A5UdxdTXs9VlJQoAR9/7KRSnjbNd/X++PuPzNk8\nx3cVFiRVW+bQmHzk0zF8EakDROIsZJ5ZKxHZKCJfikgTX563sKpUyUm0FhsLS5f6ps4KpSoU7uRq\nlSo5P3+MMT7ns2yZ6cM5HwOj03v6Ga0Haqtqkoj0AD4DGniqKy4uzvU8JiamSK0pmVn9+vDBB3DH\nHbByJTRqlMf6rqpP/avq+ya4giYCV10Fx487d9waY9yKj48nPj4+18f5ZB6+iBQHvgAWqerrXpT/\nBWiuqifc7AuKMfzMZs6E556DtWudNi+v0jSNb/Z+Q7d6hSzzZLNmzt1pzZr5OxJjCo2Cnof/LrDN\nU2MvIlUzPL8R54smS2MfzIYNg9tuc+boX7iQ9/rOJp/lvZ/eIyk5Ke+VFaTKleHYMX9HYUyR5ItZ\nOm2B/wA/A5r+eBqoDaiqTheRh4GHgGTgLPC4qrqdNB6sPXyAtDSn0a9UyUnDEJSJ1m6/HQYOdP5r\njPGKrWlbSCUmQrt2Tps3dqxv6jxx9gTlSpYrHHP0H30Umjd38kobY7xiDX4hdvCgk2tn8mS49da8\n13fvgnvp37g/vRr0yntlxpiAYw1+Iffjj9CjB3z9NURH562ulLQUiofY8sXGFFWWPK2Qu+EGePtt\n6NfPScGQFxkbe/syNSZ4WYMfwG67DUaOhL59nbH9vFq5fyVDPrOxcWOClQ3pBDhVGDrUyTjw0UfO\nUomXKzk1mcOJh6lRvobP4jPG+J+N4Rch589D587O7J3nn/d3NPlM1bnTtnJlf0diTKFhY/hFSKlS\n8OmnTgqG2bPzXt+F1AuM/HJkYK6Hm5QEtWr5OwpjiiSbulFIVK7sJFpr3x6uv97p7V+uksVK0vG6\njpQqXsp3AfpKmTJOLz8pyXlujPEZG9IpZJYsgcGDYfVqqFvX39Hkk5o1nTdoPX1jvGJDOkVU165O\nOuXeveHkybzXt+a3New5sSfvFflS5cpw9Ki/ozCmyLEGvxB66CGn4b/jjrynjt9xbAe/nfrNN4H5\nStWqcPiwv6MwpsixIZ1CKiXFmZ9fuzZMnVrEEq2NHg1t2sCdd/o7EmMKBZuWGQROn3baxREjnJxj\neZGmaew/uZ/rK13vm+CMMQXGxvCDQPnyzsydF1+Er77KW10/Hf6JJ5c86ZvAjDEByXr4RcCaNXDL\nLbBsGYSFXX49qooUqbEhY4KD9fCDSJs28Npr0KdP3q51Xmzsk1NtEXFjiqI8N/giUkNElovINhH5\nWUTcjiaLyBsisltENolIZF7Pay51113OmiG33ALnzl1+PWmaRpt32wTezB1jTJ75YonDakA1Vd0k\nIqHAeqCfqu7IUKYH8Iiq9hKRlsDrqtrKQ302pHOZVGHQICfB2ty5lz9z51jSMSqX8XMum/37oUYN\nKG43gxuTkwIb0lHVQ6q6Kf35GWA7UD1TsX7Ae+ll1gIVMi5sbnxDBGbOhL174e9/v/x6/N7YA3To\n4DT6xhif8ekYvojUASKBzAuUVwcyjhH8l6xfCsYHSpeGBQucRdDnzctbXbM3zWb2Jh9ka7sc1avn\nfeUXY8wlfPZ7OX0452NgdHpP/5Ldbg7xOG4TFxfneh4TE0NMTIwPIgwe1arBwoXQqRPUqeOsj3s5\nWtdszRXFr/BpbF6zBt8Yj+Lj44mPj8/1cT6ZlikixYEvgEWq+rqb/W8DK1T1g/TtHUB7Vc0yp8TG\n8H3niy+cm7K++865I7dQ+etf4ZprYMwYf0diTMAr6GmZ7wLb3DX26T4HBqcH1go46a6xN77Vu7fT\nXvbpAwl5SH1/POk472x4x3eBeaNGDevhG+NjvpiW2Ra4G+goIhtFZIOIdBeRB0RkBICqfgX8IiJ7\ngGnAyLye13jnscegdWtn9k5q6uXVESIhHD5zuGAXQG/UCIoVK7jzGRME7E7bIJCcDN27Q0QETJrk\n72iMMb5md9oalxIl4OOP4csvYdq0vNW16dAmUtMu86eCMcavrMEPEpUqORdxY2Nh6dLLr+fl1S+z\n6/gu3wVmjCkwNqQTZFaudBZOWbnSGSY3xhR+NqRj3Grf3kmn3Ls3HD9++fWkaRpJyUm+C8wYk++s\nwQ9Cw4bBbbdB//5w4cLl1fHP9f/kuf8859vAMjt8GHbvzt9zGBNEbEgnSKWlOY1+pUpOGobcJlq7\nkHoBQShRrET+BAhOYN9+C7Nm5d85jCkCbEjHZCskBN5/HzZtgldeyf3xJYuVzN/GHuD662Hfvvw9\nhzFBxBr8IFa2LHz+ObzxBnz66eXVcSTxCD3m9iAlLcW3wYHT4O/d6/t6jQlSNqRj+PFH6NEDvv4a\noqNzd6yqsvHQRqKvyeWB3khNhXLlnLH8cuV8X78xRYQN6Riv3XADvP029OuX+/Q1IpI/jT04qRUa\nNIAdO3Iua4zJkTX4BnAu4I4cCX37QmLi5dXxxto3OHDygG8Du+UWJzeEMSbPbEjHuKjC0KFw5gx8\n9JFzYTc35m2ZR9uabalZoWa+xGeMcc/bIR1r8M0lzp+Hzp2hXTt4/nl/R2OM8YaN4ZvLUqqUM2Pn\ngw9g9mWubngs6Rgb/tjg28CMMXlmDb7JonJlJ9HamDHOfU+5tfnQZpbsXeL7wIwxeWJDOsajJUtg\n8GBYvRrq1vV3NMYYTwp0SEdEZojIYRH5ycP+9iJyMn01rA0i8owvzmvyV9euTjrl3r3h5MnLq+No\n4tG8B7JuHWzZkvd6jAlyvhrSmQl0y6HMf1Q1Ov3xrI/Oa/LZQw85Df8dd+R+duSZC2fo9F6nvGfV\nXLQI5s7NWx3GGN80+Kq6Cvgzh2K5TM9lAsWrr0Lx4vDoo87UTW+FlgxlwwMbKFOiTN4CiIx0kv4Y\nY/KkIC/atkpf5PxLEWlSgOc1eVS8OMyb51zAnTw5l8eGFAfI2wLo1uAb4xPFC+g864HaqpokIj2A\nz4AGngrHxcW5nsfExBATE5Pf8ZkclC/vzNxp0wbq1YOePXN3/KOLHqVbvW70btA79yevWdO5QeDQ\nIahWLffHG1PExMfHEx8fn+vjfDZLR0RqAwtVNdyLsr8AzVX1hJt9NksngK1Z42Q7WLYMwsK8P+7X\nU79yTeg1l59SuWNHGDcOuuV0qciY4OOPG68ED+P0IlI1w/Mbcb5osjT2JvC1aQOvvQZ9+jhJLL1V\nq0KtvOXPf+AB5wYBY8xl80kPX0T+BcQAVwGHgVigJKCqOl1EHgYeApKBs8DjqrrWQ13Wwy8EYmOd\neforVsAVV3h/3OZDm/l679eMbTs2/4IzJshYLh2Tr1Rh0CAnwdrcud4vkXgs6Rjr/ruOHvV75G+A\nxgQRa/BNvjt7FmJioFcvmDDB39EYE7wseZrJd6VLw4IFzlrj8+bl7tg0TWPelnmkaVr+BGeMycIa\nfJMn1arBwoUwahR8/733x6VpGt/99h2nzp3Kv+CMMZewIR3jE198ASNGwHffQe3a+XSSX36B99+H\n8ePz6QTGFE42hm8K3D/+ATNnOtk1c7Pm+LGkY5QsVpLypcrnUPCYc9fX8ePOerfGGMDG8I0fPPYY\ntG7tzN5JTfX+uFfXvMpXu7/KuWDlynDttbB58+UHaUwQsx6+8ankZOjeHSIiYNIk745RVcTbeZ0P\nPQQNGsDjj19+kMYUMdbDN35RogR8/DF8+SVMm+bdMRkb+5S0lOwLx8TAZeQQMcZYg2/yQaVKzkXc\n2FhYutT74zYd2kTvf+WQXK19e/jPf3I3ZmSMAWxIx+SjlSudhVNWroRGjXIur6ocSzrG1WWvzr7g\n6tXQqpVduDUmnc3SMQFh5kx47jlYuxauusrf0RhTNNkYvgkIw4bBbbdB//5w4YJ3x6RpGkM+G8If\nCX/kb3DGBBnr4Zt8l5bmNPqVKjlpGLyZkLP8l+W0q9UubymVjQkSNqRjAkpiIrRrBwMHwljLjGyM\nT3nb4BfUEocmyJUtC59/7lxrrV8fbr3Vu+NW/7qaYiHFaFWjVdadFy5AyZK+DdSYIswnY/giMkNE\nDovIT9mUeUNEdovIJhGJ9MV5TeFSowZ89pmTc2fDBu+OOX3+NAnnE7Lu+PFH5yeDMcZrvlrx6ibg\nDPCeuzVt0xcuf0RVe4lIS+B1VXXTZbMhnWAwf76ThuH776F69cusJDkZqlSBHTugatWcyxtThBXo\nLB1VXQX8mU2RfsB76WXXAhUyrnNrgsttt8HIkdC3rzO27400TWPzoQw5dEqUgM6dYfHi/AnSmCKo\noKZlVgd+y7D93/TXTJB66ilo1gwGD3Zm8eTk11O/EhsfyyW//nr2hK+8SLpmjAEKrsF391PDxm2C\nmAhMnw5HjsAzz+Rcvk7FOnw28LNLk6x17w7ffOP9BH9jglxBzdI5CNTMsF0D+N1T4bi4ONfzmJgY\nYmJi8isu40elSsGnn0LLltCwIQwZ4t1xSclJlClRBq65Bjp0gH37vMvdYEwRER8fT/xlJBH02Tx8\nEakDLFTVMDf7egIPp1+0bQW8ZhdtzUXbtzs50ebP927iTYfZHZjcYzLNqjTL/+CMKQQK9MYrEfkX\nEANcBRwGYoGSgKrq9PQyU4DuQCIwTFXdTsyzBj84LVnijOevXg1162Zf9syFM4SWDC2YwIwpBOxO\nW1PovPUWvPGGsy5uxYr+jsaYwsOSp5lC56GHoGtXJ6VycnLO5T/c+iEvrnox/wMzpoiwHr4JKCkp\nzvz82rVh6tTsE60dPnOYC6kXqFmhpudCxgQBG9Ixhdbp09CmjZOC4dFHvTzo44+dO24t3YIJQjak\nYwqt8uWdJRJffNG7+6pOnjvJ8wfmoJPfyP/gjCnErME3AalOHafTPnQo/Pxz9mXLlChDaHRr0pZ8\nDX9ml+HDmOBmQzomoP3rX/D0084SiTnmSBswwMmv88ADBRKbMYHChnRMkXDXXc4duLfcAufO5VB4\nyBC+/2wKiRe8zMhmTJCxBt8EvLg4Z9bO8OGQ7Y+/bt34uPQv7Fz/dUGFZkyhYkM6plA4exZiYqBX\nL5gwIZuCBw86Sfa9WTjXmCLChnRMkVK6NCxY4CyCPm9eNgVr1AAR0jSNE2dPFFh8xhQG1uCbQqNa\nNVi4EEaNclbLys4n2z9h/PLxBROYMYWEDemYQueLL5ybsr77zhnbd0dVSdVUiocUVAZwY/zHhnRM\nkdW7N4wZA336QIKb9c3B+QO42NhbB8IYhzX4plB67DFo3RoGDYLUVDcFTp+GWbM4c+EMrWe0Jik5\nqcBjNCbQ2JCOKbSSk51VDiMiYNKkTDvPnXNu112+nD3VSlLvynr+CNGYAmFDOqbIK1HCSb/w5Zcw\nbVqmnVdcAY88Aq++ao29Mel80uCLSHcR2SEiu0RknJv9Q0TkiIhsSH8M98V5jalUybmIGxsLS5dm\n2vnQQ86iub87yydPXjuZH3//seCDNCZA5LnBF5EQYArQDWgKDBIRdytKz1PV6PTHu3k9rzEX1a8P\nH3wAd98NO3Zk2HHVVXDPPfD66wA0uboJVcvmlJDHmKLLFz38G4HdqnpAVZOBeUA/N+Xs1keTb9q3\nd9Ip9+4Nx49n2PHkk/Duu3DmDJ2u72SLpZig5osGvzrwW4btg+mvZdZfRDaJyIciUsMH5zXmEsOG\nwW23Qf/+cOFC+ou1ajn5lUP/b9Hzo4lHWbov8/iPMUWfLxp8dz33zNNsPgfqqGoksAyY7YPzGpPF\nCy/AlVfCgw9mSLRWrdolZQ6dOcR3v31X8MEZ42e+uA3xIFArw3YN4PeMBVQ146oU/wReyq7CuLg4\n1/OYmBhiYmLyGqMJEiEh8P77zkqHr7wCY8dmLRNWNYywqmEFH5wxPhIfH098fHyuj8vzPHwRKQbs\nBDoBfwA/AINUdXuGMtVU9VD681uBMaraxkN9Ng/f5NnBg9CqFUyeDLfe6rncgZMHqF3RQ34GYwqJ\nApuHr6qpwCPAEmArzmyc7SIyUUR6pxd7VES2iMjG9LJD83peY7JTowZ89pmTc2fDBvdlklOTuePj\nOziedNx9AWOKGLvT1hRp8+c7aRi+/95Jk8+TTzrLaEVHA06eHbHc+aaQszttjcGZtTNyJPTtC4mJ\nOJP2//pX1xXdi419mqaRpml+jNSY/GcNvinynnoKmjWDwYMhbdi9zkT9zz67pMwzy59h9iabPGaK\nNhvSMUHh/Hno3NmZvfN8h2+ctAtbt0KpUgAcTzpOhSsqWP58UyjZkI4xGZQq5aTV+eADmP17FwgL\ng5f+b3bwVWWussbeFHnW4JugUbmyk2htzBj44e7XISlrjvwdx3bw+OLH/RCdMfnPhnRM0FmyxBnP\nX70a6ta9dN/Z5LN8d/A7Ol7X0T/BGXMZvB3SsQbfBKW33oI33nDWxa1Y0d/RGJM3NoZvTDYeegi6\ndoU77nBWzspMVZm+fjpnk88WfHDG5BNr8E3QevVVKF4cHn00Q6K1DI4kHiHhgodV0o0phGxIxwS1\n06ehTRt4pvuPDLzpINxyi79DMibXbAzfGC/t3w8jWmxkYXI3Sm3Z4CTiyeBo4lGS05K5tty1/gnQ\nmBzYGL4xXqpTB+IWRPGP5FEk3HlvlvGdf2/5N1/t/so/wRnjQ9bDNybdv+ek0Oi+m6j39B2Ui/2r\nv8MxxmvWwzcmlwb9pTjxD84j+bmXOL/ye7dlbNaOKcyswTcmg8deq8M7rWbwWuyfWWbu/PLnL7Sf\n1R77BWoKKxvSMSaTs2chJgZ69YIJEy7dl3A+gXKlyvklLmM8KdAhHRHpLiI7RGSXiIxzs7+kiMwT\nkd0i8p2I1HJXjzGBoHRpWLAAZsyAefMu3WeNvSnM8tzgi0gIMAXoBjQFBolIo0zF7gVOqGp94DXg\n5bye15j8VK0aLFwIo0Y5q2VlNvjTwWw/uj3rDmMCmC96+DcCu1X1gKomA/OAfpnK9AMuri7xMc6C\n58YEtPBwmDkT+veHA3tTLtn3WKvHqHdlPT9FZszl8UUC8OrAbxm2D+J8Cbgto6qpInJSRK5U1RM+\nOL8x+aZ3b3jxL1s5Fv4QiT+spGIlZ5i0GtEcPezn4IzJLVXN0wO4HZieYfse4PVMZbYA12bY3gNU\n8lCfXny0b99eV6xYoRnFxsZqbGysZhYbG+s6ztN+O86Ou5zj0lLT9I1xB7VGDdXQ0P87LjQ0Vq+K\nXKOVWi3Qa65RveYaZ39oaKxrO+PrGY9zt9+Os+O8Pe6KK4ZoiRLtXcc7TXnO7XWeZ+mISCsgTlW7\np28/lX7ylzKUWZReZq2IFAP+UNUqHurTvMZkTEFZ//t6jiYdpXu97v4OxQQxb2fp+GJIZx1QT0Rq\nA38AA4FBmcosBIYAa4EBwHIfnNcYv2t+bXN/h2CM1/J80VZVU4FHgCXAVmCeqm4XkYki0ju92Ayg\nsojsBh4DnsrreY0JJGmaxsr9K/0dhjHZshuvglydOnU4cOCAv8Mwxnipdu3a7N+//5LXLD2y8Ur6\nPxR/h2GM8ZK7v1lLnmaMMeYS1uAbY0yQsAbfGGOChDX4JqBcd911bNu27ZLXWrRowX/+8x8AYmNj\n+eijjwosnpCQEJKSkgrsfMbkJ1/MwzemwEycOLFAzyeS43UwYwoN6+GbQmXYsGFMnToVcBr/u+66\ni169etG4cWP69OnDuXPnADh9+jS33347TZo0oUuXLgwZMoSxY8cCkJyczNixY2nVqhXR0dEMGTLE\nYy/e0wymxYsXEx0dTWRkJF26dGHv3r0A7Nq1izZt2hAVFUV4eDiTJk0CYMGCBYSHhxMdHU14eLjr\nF4sxBckafBNwbr/9dqKjo4mOjiYqKort2z2nIV6/fj3z5s1j+/btXLhwgblz5wLwt7/9jSuvvJJt\n27bx4Ycf8u2337qOefnll6lYsSLff/89GzZs4JprruH555/3Or6jR48yePBg/v3vf7Np0yYGDRrE\n3XffDcDUqVPp168fGzdu5KeffuLee+8FnKGof/7zn2zYsIHNmzcTHR19OR+NMXliQzrGI1+MZlzO\nFP/58+fTuHFj13aLFi08lu3WrRvlyjmLkrRs2dLV016xYgVTpkwBoFKlStxyyy2uYz7//HMSEhJc\n1wIuXLhARESE1/GtXbuWyMhIGjZsCDi/OkaOHEliYiI333wz48aNIzExkQ4dOtChQwcAOnXqxOOP\nP07//v3p0aMHTZs29fp8xviK9fCNR6p5f1zeeb0/8IorrnA9L1asGCkpKa46PI2/qypTp05l48aN\nbNy4ka1bt/Kvf/3LbVl3dbir++J2//79+fbbb6lXrx4vvvgif/nLXwB49dVX+ec//0mpUqUYMGAA\nM2bM8Po9GuMr1uCbIqlDhw7MmjULgD///JMFCxa49vXt25dJkya5xvvPnDnDjh073Nbj7sundevW\nbNq0iV27dgEwa9YsoqKiKFu2LHv37qVq1aoMHjyY2NhY1q1bBzhj+02bNmXUqFHcc889rteNKUg2\npGMCirsedcbXvJ01M2HCBIYPH06TJk245ppraNGiBRUqVADgqaeeIi4ujhYtWhASEkJISAixsbE0\napR5ZU7nfA0bNnTdzh4aGsr27dt57733GDRoEKmpqVx99dW8//77AHz44YfMnTuXkiVLEhISwhtv\nvOE65549eyhWrBiVKlWyHr7xC8ulE+SKai6dlJQUUlNTKVWqFAkJCdx000384x//oGPHjv4OzZg8\nyUsuHevhmyLpzz//pEePHqSmpnL+/Hnuvvtua+xN0LMefpArqj18Y4oqy5ZpjDEmR3lq8EWkkogs\nEZGdIvK1iFTwUC5VRDaIyEYR+Swv5zTGGHN58jSkIyIvAcdV9WURGQdUUtUsyxeKyGlVLe9lnTak\nU4BsSMeYwiUvQzp5bfB3AO1V9bCIVAPiVTXL3DYRSVDVcl7WaQ1+AbIG35jCxZ9j+FVU9TCAqh4C\nrvZQrpSI/CAia0SkXx7PaYqwlJQUJkyYQMOGDYmMjKR58+aMGTOG1NTUAo2jTp06NGnSxJXPJzo6\nml9//TXH4yZOnOi629cfDhw4wNVXe/ozNMEux2mZIvINUDXjS4ACz+TiPLVU9ZCIXAcsF5GfVPUX\nT4Xj4uJcz2NiYoiJicnFqUxhNnToUM6fP8/GjRspU6YMaWlpvPvuu5w/f54yZcpcUjYtLY2QkPyZ\ndyAiWXL6eGPixImMGTOG4sWz/mmlpqZSrFgxX4XokaV0Lvri4+OJj4/P/YGqetkPYDtQNf15NWC7\nF8fMBPpns19NwQmkz3v37t0aGhqqp06dcrt/1qxZ2rlzZ7311ls1LCxMN2/erHv27NFOnTppeHi4\nNm/eXBcMsVwgAAAd5ElEQVQvXqyqqklJSTpgwABt2rSpRkZG6p133qmqqjt37tTWrVtrZGSkhoWF\n6auvvur2XHXq1NGtW7e63Sci+vzzz2uLFi20bt26+sknn6iq6sMPP6whISEaERGhUVFReurUKR06\ndKjed9992q5dO42KilJV1UWLFmlUVJRGRERo586dde/evaqqGh8frxERETp48GBt2rSptmzZUrdv\n366qqj179tT58+e7Ypg/f75269YtS2z79+/Xq6++2m3cs2fP1rCwMI2IiND+/fvr0aNHVVV1zZo1\nGh0drVFRUdqsWTOdN2+eqqpOmzZNGzdu7Ip1586dbus1Bcvd32z6azm32d4U8ngwvASMS38+DnjR\nTZmKQMn055WBnUCjbOr07adjshVIn/eHH37oahTdmTVrlpYrV05/+eUX12stW7bUmTNnqqrqtm3b\ntHLlynrs2DH99NNPtXv37q5yJ0+eVFXV0aNH64svvpjl9czq1KnjauwiIyO1RYsWrn0iolOnTlVV\n1dWrV2v16tUv2ZeUlOTaHjp0qLZo0ULPnj2rqqpHjhzRq6++Wnfs2KGqqjNmzNCWLVuqqtPgh4SE\n6LfffquqTgN9ww03qKrq4sWLtUOHDq56O3XqpAsXLswSt6cGf8uWLXrttdfq4cOHVVV1/PjxOnDg\nQFVV7devn6uRV1XXF26FChX00KFDqqp64cIF13sw/uXPBv9KYGl6I/4NUDH99ebA9PTnrYGfgI3A\nZmBoDnX6/AMynmX7ecfGqtskmLGx3pX3VM6DDz74IMcGv0ePHq7thIQEveKKKy4p06VLF/3iiy90\n3759Wrt2bX3kkUf0o48+cjXC8+fP13r16un48eN1+fLlHs9Vp04d3bZtm9t9IqLHjx9XVdXU1FQV\nET1//rxrX2Jioqvs0KFD9aWXXnJtL1y4ULt06eLaTktL01KlSumZM2c0Pj5eGzRokGVfQkKCqqo2\nadJEd+zYoTt27NDrrrtO09LSssTmqcGfPHmy3n///a7tgwcPauXKlVVV9bXXXtNmzZrps88+q2vX\nrnWV6d+/v3bt2lUnT56s+/bt8/hZmYKVlwY/TwOgqnpCVTurakNV7aKqJ9NfX6+qI9Kff6eq4aoa\npaoRqjorL+c0BSguzn3O4wzXWLIt76mcB9HR0ezevZtTp055LBMaGup67vw7v5Sqk7r4uuuuY+vW\nrXTp0oWlS5cSERHBhQsXPKYvdsdd/eCMkV9MyxwSEoKIZHuhNnPMnlIrezrXRQ8//DBvvvkmb775\nJg888ECuxuqzO+/o0aP5/PPPqVKlCqNGjWL8+PGAsy7Bc889R1JSEh06dODrr7/2+nwmQHnzrVCQ\nD6yHX6AC7fO+66679M4773T1alNSUvSdd97RxMREnTVrlg4YMOCS8q1atdJZs2apqur27du1SpUq\neuzYMT148KCrV5+YmKiVKlXSP/74Q/fs2ePqGa9evVobNmzoNo6cxvAz9uIzbleoUEF///13176h\nQ4fqm2++6do+evSoVqlSxTUe/u6772rr1q1V9f+GdFatWqWqqnPmzLlkKCkhIUFr1qypVatW1WPH\njrmNbf/+/a6ee0Zbt27VGjVquIZ0JkyYoIMGDVJV1V27drnKzZ07V7t166apqamuawuqqvfff7++\n8MILbs9pCpa7v1m87OFb8jQTUGbPnk1cXBzNmzenVKlSpKWl0bNnT0qVKuW2/Ny5cxkxYgSTJk2i\nRIkSvP/++1x11VUsXryYp55y7gFMS0vj6aefplq1arzwwgtu0xdnJiLcfvvtXHHFFa7e8TvvvEN0\ndHS2PfQnnniCDh06UKZMGeLj47OUrVy5MnPmzHGbWhkgPDycd955hwcffJCyZcvy3nvvufaFhobS\nvXt3zp07x1VXXeXxMzx58iS1atUCnA5d48aNWbJkCc8//zydO3cmJCSE66+/nmnTpgHwxhtvsGLF\nCkqWLMkVV1zBlClTSElJYejQoZw6dQoRoVatWrz00ksez2kKB0ueFuTsxqvAsXLlSsaMGcMPP/zg\ndn9KSgoRERG89957NG/evICjM4HCkqcZU8QtXLiQevXq0b17d2vszWWzHn6Qsx6+MYWL9fCNMcbk\nyBp8Y4wJEtbgG2NMkLAG3xhjgoQ1+MYYEySswTcBxfLh511sbCzNmjUjMjKSZs2a8dprrwGwfv36\nbFNJ5EccH330kdt9EydOZOzYsW73XXfddWzbti0/QwtadqetCSiWDz9vPv74Y1asWMHGjRspUaIE\nycnJ7N27F4DmzZszZ86cfD1/RhMnTiywcxnvWA/fBIw9e/awYMECZsyY4WrcQ0JCuO+++yhTpgyz\nZ8+mS5cu9O/fn/DwcLZs2cLevXvp3LkzERER3HDDDa4EX2fPnuWOO+6gWbNmREVFMXDgQAB27dpF\nmzZtiIqKIjw8nEmTJnmMx9P9CSEhIbzwwgvceOON1KtXj08//RSARx55BBGhTZs2REdHc/r0aYYN\nG8b999/PzTffTIsWLQBYvHgx0dHRREZG0qVLF/bt2wc4d9pGRkYyZMgQmjVrRqtWrdixYwcAvXr1\n4pNPPnHF8Mknn9C9e/cssR08eJDKlStTokQJAEqUKEGjRo1c9V+MAWDKlCk0aNCAli1bEhcX51op\n6+KqWU8//TTR0dE0adKEDRs2MGLECCIiImjdujVHjhwBnC/dJ598krCwMMLDwxkzZozrcxs2bBhT\np04F4PTp0wwYMIAmTZrQsWNH15dQbqxbt442bdoQGRlJ27Zt+fHHHwE4evQoXbp0ISIigoiICJ54\n4gkA1qxZQ/PmzYmOjiYsLIwPPvgg1+cscrxJuFOQDwIsmVdRF0ift+XDz3s+/D/++EMbNmyo9evX\n12HDhun777+vKSkprvovvo/NmzdrjRo1XGmeH3vsMVda5f3796uI6KJFi1RV9ZVXXtGKFSvqTz/9\npKqqI0eO1PHjx6uq6tSpU7VLly6akpKiycnJ2qlTJ3377bdd7/1i4rgnnnhC7733XlVVPXbsmNaq\nVUvHjBnj8bPPnLjuwoULWqtWLVdK62XLlmmtWrU0OTlZ//GPf+iDDz7oKnvx/6mnPP+Fnbu/WQoi\nPbIp2uLi4hARROSSZScz7vf0uqdjsqNe3PF70003UadOHQDOnDnD5s2bGTp0KACNGzcmKiqK77//\nnoiICLZv386oUaP4+OOPKVmyJAA333wz77zzDhMmTGDFihVUqFDB47nmz5/Phg0b2LhxY5b8Nnfe\neScArVq14vfff+fChQse38fFJGwAa9euJTIykoYNGwJOL3jTpk0kJiYCUK9ePW666SYA/vKXv/Dz\nzz9z5swZunXrxuHDh9m5cyc7d+5k37599OrVK0vM1apVY9u2bcycOZOGDRvy/PPP06dPnyzlVq5c\nSc+ePbnyyitdcWRUrlw51y+I6OhoatasSVhYGOAMDe3ZsweAZcuWMXToUIoVK0bx4sUZNmwYS5cu\nzXK+FStWcO+99wJw1VVX0b9//yxlsrNz505KlSpFhw4dAOjYsSOlSpVi586dtGrVikWLFjFu3Di+\n/PJLypYtC0CHDh149tlnee655/jhhx8oX758rs5ZFFmDbzyKi4tz9Qxy2+B7OiY7lg/f/bku8jYf\nfkhICG3btmXcuHGsWLGCxYsXc/LkySzvLePxmd9rxuykxYoVc73fi9sX36+378ebL/PsuDvPxdda\ntWrFxo0bXdcoLn4pZM7zP2HChDzFUBTkqcEXkdtFZIuIpIpIdDbluovIDhHZJSLj8nJOU3TVq1eP\nvn378sADD3DmzBnAudA5Y8YMkpKSspQvV64ckZGRzJ49G4AdO3bw008/0bJlS/773/8SEhJC3759\nmTRpEseOHePEiRPs3buXqlWrMnjwYGJjY1m3bl2u48zceGXcLl++fLZfWK1bt2bTpk3s2rULgFmz\nZhEVFeXqle7Zs4fVq1cDTurnsLAw1xfG4MGD+eyzz/jwww+577773Na/YcMGDhw44Npev349V155\nJRUrVrykXExMDF999RXHjx8HuCQNs7v36EmXLl2YNWsWKSkpJCcnu66zZNapUydmzpwJwPHjx13X\nPbzVqFEjzp8/z8qVKwHnF0NKSgoNGjRg//79lCtXjjvuuINXX32VDRs2ALB7926uu+467r//fkaP\nHu0xC2kwyessnZ+BW4FpngqISAgwBegE/A6sE5EFqrojj+c2RZDlw89bPvxjx44xcuRIEhISKFmy\nJGXLlmXBggVZyoWHhzN27FjatGlD+fLl6dix4yXDW96upjVixAj27t1LVFQUIkL37t1dX0YZ6xg/\nfjzDhw+nSZMmVKtWjfbt23usU0To3LkzxYsXd332P//8M/Pnz2fUqFEkJSVRtmxZ5s+fT/HixYmP\nj+fVV191lfeU53/y5MlevaeizCfZMkVkBfCEqm5ws68VEKuqPdK3n8K5wOB2NQXLllmwLFtm4Cjo\nfPhnzpxx/XqYOHEie/fuzdLTN4En0LNlVgd+y7B9MP01Y4yX8iMf/lNPPUVUVBRNmzZl/fr1vPzy\nyz6p1wSuHId0ROQboGrGlwAF/p+qLvTiHO6+dbLtUma82BcTE0NMTIwXpzGmcGvfvr3H3n2fPn3c\nzrbJiylTpvi0PlNw4uPjiY+Pz/VxBTWkE6eq3dO3bUgngNiQjjGFS6AM6Xg62TqgnojUFpGSwEDg\ncx+e1xhjjBfyOi3zFhH5DWgFfCEii9Jfv0ZEvgBQ1VTgEWAJsBWYp6rb8xa2McaY3LI1bYOcDekY\nU7gEypCOMXkyfvx4Ro4c6dr+4osvCAkJYfv2//tB2KdPH9cNPJ54mwb4YpIwd06dOsUrr7ziZeRZ\nZZfid9euXfTv35969epx44030q5dOz7/vGBHOVeuXEnZsmUvSf98MUVFdjZv3uwx5XFByS61cn55\n4403sk20d9GRI0fo1q0bDRs2JCoqyuNF+NmzZ1OpUiXX53/bbbe59o0bN4558+b5LPZLeJNwpyAf\nBFAyr2AQSJ/30qVLtUmTJq7tMWPGaOvWrV2JylJTU7VixYqXJE/Li/3797sShmX2yy+/aOXKlS+7\n7uuuuy5LAjBVJ7lZtWrVdO7cua7XDh8+rHPmzMlS9mLSs/yQMZFabsycOVNvv/12j/vzM+aL4uLi\nPCZeyw9JSUlat25dPXfuXI5lhw8frs8995yqqq5atUrr1avnttysWbN0wIABbvcdPXpUGzZs6PEc\n7v5mseRpprBp27Ytv/zyC0ePHgWcXugzzzzDihUrACdtQIUKFVzJ0xYtWsRNN91EixYtaNu2LWvX\nrnUd500aYHA6PM888wzR0dE0btyYNWvWAE6q41OnThEdHe1KZnbo0CEGDBhAq1atiIiI4MUXX3TV\n8+233xIeHk5ERASjRo3yOEz25ptv0rFjR+666y7Xa1WqVOGee+4BnIRfjz/+OK1bt6Zfv36Ak/Yg\nPDycyMhIbrvtNo4dOwbAd9995zb97/Tp012Lt0RGRrrSOHhr9uzZdOvWjYEDB9KsWTPatWvHkSNH\nOHHiBLGxsSxbtozo6Ggee+wxwMndM3HiRG688Ub+9re/5ZgyecSIEbRt25ZGjRrxwAMPkJKSwh9/\n/MG11157SRK6fv365aqnm5iYyPDhw13nzXhfwcSJE12fSfPmzTl9+rTHFNqZzZ8/n/bt27vu9n72\n2WddPfKkpCTCwsJYvHgxAB9++CEPPvgg4Px7Ll26NOvXr3dbr6d/I5UrV6Zu3bosW7bM6/fuNW++\nFQryQQD1OINBoH3eN998s3744YeakJCgzZo107S0NK1fv76qqv7v//6vDh06VFVV9+7dq61bt9aE\nhARVVd26davWqlVLVXOfBvirr75SVdW5c+dq27ZtXfsy9/67dOniSl184cIFbdeunS5dulTPnz+v\n1atX1//85z+q6qR5DgkJcdvD79mzp77++use339MTIz269dPU1NTVVV1y5Yteu211+rhw4dVVXX8\n+PE6cOBAVfWc/rdChQp66NAhV5wXUzNnFB8fr2XKlHGlf46KitK///3vqur0Pq+88kr973//q6qq\n999/vz7zzDOufZl7piKir7zyimv7rbfeyjZlckREhCYlJWlqaqp27drVlUJ50KBB+t5777k+/+rV\nq2tycnKW2D318MeNG+f693H69Glt2rSpLl68WP/880+tWLGiq4d+5swZTUlJ8ZhCO7N7771Xp02b\n5tpOS0vTbt266eTJk3X48OH61FNPqarq8ePHNTQ09JJje/bsqZ9++mmWOmfNmqVVqlTRqKgobd++\nvX755ZeX7P/b3/6m//M//+M2Hnd/s1gP3+RVXHwccfFxl719OWJiYoiPj2fVqlXcdNNNiAgNGjRg\n27ZtxMfHu27C+/rrr9m3bx8333wzUVFR3H333aSlpbl+HVzkTRrgHj16AE6q44uLkWSWlJREfHw8\njz76KFFRUdx444388ccfbN++nZ07d1K2bFnatWsHwIABAzym4lUvLpDfddddrpW8VqxYQa9evahS\npQoADzzwgCv9sKf0v506dWLw4MFMmTKFgwcPXpLpMqOmTZu60j9v2LCBZ555xrWvbdu2XHvtta7P\nJacFSwYPHux6vnTp0mxTJt95552ULl2akJAQhgwZwvLlywEYNWqUa8GUt99+m+HDh7tdOcyTpUuX\ncv/99wPO/9dBgwaxdOlSypcvT/369Rk8eDDvvPMOCQkJFCtWzGMK7cwOHjxI1ar/d++piDBnzhxe\neOEFtm/fznPPPed1jBf16dOHX3/9lQ0bNvDaa68xfPhwdu7c6dpfrVo1Dh48mOt6c2JLHBqP4mLi\n8rR9OWJiYnj44YcpX768K8HWzTffzPLly1m9ejVvvvkm4DSc3bt3Z9asWdnWp7lMA+wpzfHF5RR/\n/PHHLMsq/vTTT16/v+bNm7uGnjzxNp3y6NGj6du3L0uXLmXUqFF07dqVv//978yfP58ff/yR5cuX\n06FDB6ZNm0a3bt28jhHwmA7ZHRG57BTQGcu2bt2a1NRU1qxZw+zZs3OdydTTeUNCQvj+++9ZvXo1\ny5Yto3nz5nz99dc0a9aMrVu3smzZMr766iuefvpptmzZkqXhL126NOfOnbvktX379hESEsKff/5J\nUlISoaGhrk7FiRMnXM9//fVXatasmSXWi/sBIiMjuemmm/jhhx9c6yScO3eO0qVL5+r9e8N6+Cag\ntGnThv379/PJJ5+4evPt2rVj8uTJVKxYkVq1agHQtWtXFi9efMlMmItL3mWU2zTAF7fLly9PUlKS\na/H00NBQ2rVrx/PPP+8qe/DgQY4cOUKjRo04e/Ysq1atApx1ZU+fPu32/Y0cOZLly5dfMjZ99OjR\nSzJmZtSpUye++uor15KC06dPp3PnzkDW9L/r1q0jLS2Nffv2ccMNNzB27Fi6du3Kxo0b3dbtza+N\nzNylf85cT04pkz/66CPOnj1LSkoK77//vit/PTjXTgYOHEjbtm2pXt1zyi13sXfp0oV33nkHgISE\nBObNm0fnzp05c+YMR44coV27dsTFxdGsWTO2bNniMYV2ZmFhYZf0vv/880/uuecePvjgAwYOHOj6\nVQHOr7u33noLgFWrVnHu3Dm3uY9+//131/MDBw6wdu1awsPDXa9t376diIgIj+//snkz7lOQDwJs\nTLmoC8TPOyYm5pJZCsnJyRoaGqrDhw+/pNw333yjrVq10sjISG3SpInef//9qpp1BsrkyZO1QYMG\nesMNN+jYsWNdMycyj9Nn3h4xYoQ2btzYNa5/6NAhHTRokIaHh2tYWJi2bdtWd+7cqarOjIywsDCN\niIjQUaNGuV2m76KdO3dqv379tG7duhoeHq5t2rTRzz//XFVVO3TokGU8d86cOa66b731Vj1y5Iiq\nqj7yyCPatGlTjYqK0tatW+v69ev1/Pnz2q5dOw0PD9eIiAjt06ePnjhxIksM8fHxWrZsWY2KinKN\n4/fq1UtVs47TZ9w+deqUtmnTRiMjI3X06NGqqhoSEqKJiYmu8qmpqfrkk09qs2bNNCwsTMeMGaNp\naWmq6ozhjxgxQtu2basNGjTQBx544JJx+vPnz2toaKhrKUN34uLitHz58lqzZk2tUaOG1qxZU6dP\nn66JiYk6dOhQ13kvXlc4ePCgtmzZUiMiIrRZs2Y6ZMgQPX/+vC5atEgjIiI0IiLikvKZbdy4UVu3\nbu3avvXWW13LZKampmqHDh1cY/yHDh3Szp07a/369TUiIkK///5713H33Xefa1nKp59+Wps2baqR\nkZEaGRmZZZZWgwYN9ODBg27jcfc3i5dj+HbjVZALhhuvLA1w4Bg2bBgtWrS45H6LjFatWsXIkSNz\nNUxWEHr06MGLL76YP73uTJYsWcLcuXNdC/tklpcbr2wM3xR5Tz31FKtXr+bChQvUrVuX6dOn+zuk\noJXdwir33XcfS5cuZc6cOQUYkXemTJnC7t27C6TBT0hI4KWX3OaWzDPr4Qe5YOjhG1OUWGoFY4wx\nObIG3xhjgoQ1+MYYEyTsom2Qq127drYX0owxgaV27dqXfWyeLtqKyO1AHNAYaKFuljhML7cfOAWk\nAcmqemM2ddpFW2OMyYWCumj7M3ArsDKHcmlAjKpGZdfYFxaXs3iwP1icvhUMca7cv5KtR7b6Lphs\nBMPnGWjy1OCr6k5V3Y3n9WwvkryeK5AUln8AFqdvBUOcvyf8zrGkY74LJhvB8HkGmoIaw1fgaxFR\nYLqq/rOAzmuMyYVBYYP8HYLJRzk2+CLyDVA140s4Dfj/U9WFXp6njaoeEpGrgW9EZLuqrsp9uMaY\ngpCalsruE7tpVLmRv0MxPuSTO21FZAXwhKeLtpnKxgIJqup2gcj0XwHGGGNyoaBz6bg9mYiUAUJU\n9YyIlAW6AhM9VeJN0MYYY3IvTxdSReQWEfkNaAV8ISKL0l+/RkS+SC9WFVglIhuB74GFqrokL+c1\nxhiTewGXPM0YY0z+CLipkiLyNxHZLCIbRWSxiFTzd0zuiMjLIrJdRDaJyHwRcb+IqZ+JyO0iskVE\nUkUk2t/xZCYi3UVkh4jsEpFx/o7HHRGZISKHRSSwkrRnICI1RGS5iGwTkZ9F5FF/x+SOiJQSkbXp\nf98/p1/TC1giEiIiG0Tkc3/H4omI7M/QZv6QbdlA6+GLSKiqnkl/PgpooqoP+TmsLESkM7BcVdNE\n5EWcFWf+x99xZSYiDXFufJsGPOnNhfWCIiIhwC6gE/A7sA4YqKo7/BpYJiJyE3AGeE9Vw3Mq7w/p\nHaNqqrpJREKB9UC/QPsswbmup6pJIlIMWA08qqrZNlT+IiKPA82B8qra19/xuCMi+4DmqvpnTmUD\nrod/sbFPVxansQo4qrpUVS/G9j1Qw5/xeJKLm+P84UZgt6oeUNVkYB7Qz88xZZE+hTjHPyZ/UtVD\nqrop/fkZYDvgeVFYP1LVpPSnpXAmjgRWrzOdiNQAegLv+DuWHHh9Y2vANfgAIvKsiPwK3AVM8Hc8\nXhgOLPJ3EIVQdeC3DNsHCdBGqjARkTpAJLDWv5G4lz5MshE4BHyjquv8HZMH/wDGEKBfSBlcvLF1\nnYjcn11BvzT4IvKNiPyU4fFz+n/7AKjqM6paC5gLjPJHjN7EmV7m/+EkhPtXIMcZoNz96gj0P66A\nlj6c8zEwOtOv5YChqmmqGoXzq7iliDTxd0yZiUgv4HD6ryYhMH8hX9RGVW/A+TXycPoQpFt+SY+s\nql28LPpv4EucjJwFLqc4RWQIzofcsWAici8Xn2egOQjUyrBdA2cs31wGESmO09jPUdUF/o4nJ6p6\nWkTige7ANj+Hk1lboK+I9ARKA+VE5D1VHeznuLJQ1UPp/z0qIp/iDJW6zWQQcEM6IlIvw2Y/nLHI\ngCMi3YGxQF9VPe/veLwUaL2UdUA9EaktIiWBgUCgzoYI9F4ewLvANlV93d+BeCIilUWkQvrz0kBn\nIOAuLKvq06paS1Wvx/l3uTwQG3sRKZP+q44MN7Zu8VQ+4Bp84MX04YhNOP8YRvs7IA8mA6E4uYE2\niMhUfwfkjqeb4wKBqqYCjwBLgK3APFUNuC94EfkXsAZoICK/isgwf8eUmYi0Be4GOqZPz9uQ3ikJ\nNNcAK9L/vtcCX6vqV36OqTDL1Y2tATct0xhjTP4IxB6+McaYfGANvjHGBAlr8I0xJkhYg2+MMUHC\nGnxjjAkS1uAbY0yQsAbfGGOChDX4xhgTJP4/w3EOWldj7xwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fb9e4356e10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot the output\n",
    "x_array = sess.run(x_vals)\n",
    "plt.plot(x_array, hinge_y_out, 'b-', label='Hinge Loss')\n",
    "plt.plot(x_array, xentropy_y_out, 'r--', label='Cross Entropy Loss')\n",
    "plt.plot(x_array, xentropy_sigmoid_y_out, 'k-.', label='Cross Entropy Sigmoid Loss')\n",
    "plt.plot(x_array, xentropy_weighted_y_out, 'g:', label='Weighted Cross Entropy Loss (x0.5)')\n",
    "plt.ylim(-1.5, 3)\n",
    "#plt.xlim(-1, 3)\n",
    "plt.legend(loc='lower right', prop={'size': 11})\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Softmax entropy and Sparse Entropy\n",
    "\n",
    "Since it is hard to graph mutliclass loss functions, we will show how to get the output instead"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 1.16012561]\n",
      "[ 0.00012564]\n"
     ]
    }
   ],
   "source": [
    "# Softmax entropy loss\n",
    "# L = -actual * (log(softmax(pred))) - (1-actual)(log(1-softmax(pred)))\n",
    "unscaled_logits = tf.constant([[1., -3., 10.]])\n",
    "target_dist = tf.constant([[0.1, 0.02, 0.88]])\n",
    "softmax_xentropy = tf.nn.softmax_cross_entropy_with_logits(logits=unscaled_logits,\n",
    "                                                           labels=target_dist)\n",
    "print(sess.run(softmax_xentropy))\n",
    "\n",
    "# Sparse entropy loss\n",
    "# Use when classes and targets have to be mutually exclusive\n",
    "# L = sum( -actual * log(pred) )\n",
    "unscaled_logits = tf.constant([[1., -3., 10.]])\n",
    "sparse_target_dist = tf.constant([2])\n",
    "sparse_xentropy =  tf.nn.sparse_softmax_cross_entropy_with_logits(logits=unscaled_logits,\n",
    "                                                                  labels=sparse_target_dist)\n",
    "print(sess.run(sparse_xentropy))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
